You're underselling it. These single-family zoning rules were created explicitly by racists whose stated goal was to keep black people out of their communities.

See e.g. The Racist History of Single-Family Home Zoning, https://www.kqed.org/news/11840548/the-racist-history-of-sin...