Might it be at least somewhat lower level than that? I'm thinking along "selfish gene" lines. You are more likely to share genes with people that look like you than with people that look nothing like you, and so favoring those who look like you should help copies of your genes to make it into the next generation. A genetic predisposition against people who look too different could be selected for.