There could be, but I think the presumption should be that it is normal social evolution to stigmatize minorities. Red-headed people, Jewish/Semitic people, people with cleft palates, on and on.

There seems to be a representation threshold below which any observable minority is considered undesirable. To the extent there is a “legitimate reason” it is probably rooted in evolutionary psychology for avoiding too much genetic variation in small tribes. I submit those fences are the opposite of Chesterton’s fence and can happily be ignored in today’s society.