When you have more people, it gives everyone more data points to separate the individual from the group.

But when you are a single data point that represents a group, any personal characteristic can be easily construed/extrapolated as a characteristic of the group you are supposed to represent.

This leads to the feeling of "Am I being judged for myself or for my group?". This cuts for both the good and the bad stereotyping that your group might have.