This isn't an activity many people have to do often (verbally classifying facial features), so it isn't that surprising to me that untrained people have difficulty with it. Obviously I don't know your specific experience, but I'd think your experience is the norm rather than the exception.

In my mind, it's similar to musicians training solfege or chemists training in verbally classifying a smell: it aids both their understanding and their skill. It absolutely blew my mind to have a perfumist friend of mine describe smells to me as we were walking through the city. I'd never have guessed the depth and complexity of olfactory sensations that us humans are capable of with the right training.