I imagine it thus: Human appearance is a vector space in our minds and we tend to put ourselves at the origin of the vector space. When we look at two people who look like us, the vectors that describe their appearance may be comparatively quite different because of their closeness to us. But if we look at two people of the same, different race, their appearance is quite removed from us, so the appearance-vectors that lead from us to them will be comparatively close together in direction and have comparatively similar magnitudes.

A Japanese friend of mine once told me I looked like a celebrity -- Manuel Schwarz, an Austrian model who was once married to Ayumi Hamasaki. Near as I can tell the only similarities between my appearance and his are that we are both white and we both have brown hair.