Well, I wouldn't deem it that unconceivable. It's a somewhat sad, but well known fact that we do narrow this "in-group" even more to the effect that we are not that good at identifying individuals of ethnic groups, we're not so familiar with. It may be well that we start with a broad, hard-wired concept and refine this by adaptation to the given environment. (It may be interesting, how developments would go with an actual "wild child".)

Edit: I always liked to think that the stereotypical alien face somewhat resembles this blueprint (plus excessive light bleed as we must have experienced it at an early age).