Probably each person has some mind-image that’s more common than others (probably guided by early childhood experiences, or mirror image of you), or even if you’re on average unbiased there’s still going to be some image you have in mind for each particular person, which is a perfectly legitimate reason to be surprised upon finally meeting that person.
Honestly the only reason this is controversial is because of “skin color” and the current political climate.
“Oh, I’m surprised, I imagine you having red hair.”
“How come you imagined me having red hair?”
“Don’t know, just because.”
Inherent in assuming a skin color is the implied association between, for example, intelligence and skin color.
"I'm surprised you're black," is not so far off from the acknowledged insult, "You're really $X for a woman."
WRT your example, anger issues are stereotypically associated with red hair, so there could be an implied "you have anger issues" that goes with that. "Oh, no reason," is also a very common deflection when being called out for poor assumptions or implied insults.
There is always a reason associations between mental traits and physical features exist, even if you're not consciously aware of it.
EDIT: And to go a different direction, if someone truely doesn't make such associations... Now that they know some people do recognize those associations, just be polite and don't make such comments about people's appearance - be it skin color, hair color, height, weight, etc.
If you’re making the comment, it’s because you’ve internalized a few stereotypes around skin color, and made an assumption based solely on those stereotypes. Stop yourself at the point where you’re making the assumption.