Also humans self-segregate. All else being equal a human who has property A will prefer to live with other humans who have property A, rather than humans with property B. Skin colour, favourite icecream flavour, late risers versus early birds - anything. We can deliberately make adjustments at policy level to reduce the harmful consequences (or of course, we can do stuff like red-lining to make it worse) but that's what they naturally do.

So these cultural affects will snowball.