> how much people can borrow

This is the main factor, and it's a result of the monetary policy of the last decades.

Since this number has been increasing compared to wages, it's becoming less sensible to build apartments for rental. But at the same time, those are needed to increase density and availability of affordable housing in attractive cities.

It's a mess.

It's complicated. The loose monetary policy is certainly one aspect. Another is that most American city-center residential areas are still zoned at low densities. If organic growth could lead to organic densification in central locations, there would be less impact to the housing market, but American zoning is stuck in the aftermath of WWII and is loathe to densify.

Yep. One study^1 estimates that zoning and land use restrictions across the country leads to an (unnecessary) increase in housing costs worth 2% of the US GDP (about $400 billion).

[1]: https://pubs.aeaweb.org/doi/pdfplus/10.1257/jep.32.1.3

Environmental regulations and such also artificially increase the price of older buildings because you can buy existing buildings that don't fulfil new regulations but can't build new ones. This is effectively a top-down supply restriction of low quality housing.