Just to keep up with the national US growth rate (0.7%), San Francisco alone would need to make housing for almost 6000 new residents per year. In the first decade of this century (2000-2009) there was a total of 2,500 added per year, or roughly a half of what's needed, and that's a simplistic calculation for what's needed.
Add in a renewed popularity of cities and urban living over the suburbs, and it's no wonder then, that housing is in short supply in San Francisco.
The reality that San Francisco doesn't want to face is that housing for at least 6000 people needs to be added EVERY SINGLE YEAR, just to keep up. All the challenges related to adding housing in San Francisco (here's a hint; a 150-year old facade isn't historic) add up to one thing - very limited supply, and any Econ 101 student could tell you the result. What happens when demand rises while supply stays the same (or even goes down, relatively)? Prices skyrocket.
Eh, anything older than 1906 in SF is historic.
My view is slightly more nuanced. Developing is expensive. Luxury properties, like all luxury goods, sell at a higher margin than non-luxury. Therefore developers will build higher margin luxury apartments until demand for luxury apartments is satisfied. At that point they'll build mid-tier. Which have lower margins but it's still profitable and better than not making money. Then, once the mid-tier is finally satisfied, the low end will be built. Margins are likely crappy but hey whatever.
Once you reach full land use there's another layer of complexity. You can't build out so you have to build up. And building up is more expensive. In fact each floor is more expensive than the one below it. It doesn't take long for simply breaking even to be too expensive for even the lowest tier when land costs come into play.
Low income developments only happen as mandated by local municipalities, and even so developers do so reluctantly only to qualify for the tax breaks on their larger maximum market rate brand new condo in the freshly gentrified neighborhood developments.