Once upon a time, neighbors also collectively agreed to zone by race and ethnicity, until Buchanan v. Warley in 1917 put an end to that practice and created the minimum-lot-size SFZ proxy system we have today. You can argue persuasively for respecting SFZ, but you probably can't do it from a moral high horse (which is just to say: you can't rest your argument on a moral right).

I’m not sure the implication that those of us who don’t enjoy sharing walls are immoral racists is particularly justified.

It's a historical fact. I'm not saying you're a racist. Those zoning ordinances though definitely are.