We can't have density without multi-family units. And density is what's required to make housing affordable in medium-size or larger cities.

> must be either paying for a ritzy place

Yes, a compromise for a more pleasant place to live is price. But a side-effect of more affordable housing is that nicer places also get cheaper.

Another compromise to get a nicer place to live for the same price is to move to less desirable location.