And there's the crux of the problem. You want to base housing policy on your arbitrary feelings, not on things that actually make for good and successful housing policy.
> This is what allows many people of different backgrounds and perspectives to coexist peacefully in a nation, with local communities that are more tightly knit
I'm sure you aren't intending this, but that's just another way of saying that segregation and red-lining should be legal. Because that's the end result when you allow these types of decisions to be made at a local level, without allowing interference from a county, state, or federal government.
The way for people of different backgrounds and perspectives to coexist peacefully is to refuse to allow subsets of those people to impose their values on others.