And there's the crux of the problem. You want to base housing policy on your arbitrary feelings, not on things that actually make for good and successful housing policy.
> This is what allows many people of different backgrounds and perspectives to coexist peacefully in a nation, with local communities that are more tightly knit
I'm sure you aren't intending this, but that's just another way of saying that segregation and red-lining should be legal. Because that's the end result when you allow these types of decisions to be made at a local level, without allowing interference from a county, state, or federal government.
The way for people of different backgrounds and perspectives to coexist peacefully is to refuse to allow subsets of those people to impose their values on others.
If you believe in locality of power, why stop at the town level? The property owner is the most hyperlocal possible unit, so they should have the right to build whatever they want on the land they own. They are local to their own property, their busybody neighbors are not.