I've frequently thought about it in terms of whether people consider the "average" stimuli to be potentially harmful or harmless - something that would take seed early in someone's childhood (with potential genetic factors as well). An "introvert" or "extrovert" would just be a heuristic to define someone's general state of mind - I know many introverts who have extroverted personalities in certain situations, and many extroverts who act like introverts in other situations.