And as another commenter mentioned, the current density of SF is incredibly low. One and two story buildings cover much of the area. Even if it just looked like Paris (wall to wall 5 story buildings), SF could house many times the number of people it currently does.
Zoning rules and NIMBYism are the main reasons for the housing crisis.
Building permits and zoning laws definitely cause a lot of problems, but the trend of housing being unaffordable for young people in cities seems to be happening everywhere. I think there are other problems at play as well, and one of them is simply housing density vs job density. Jobs are often more densely packed than housing and this will drive up prices.
I live in the SF Bay Area, and I've spent a decent amount of time studying this problem. These debates are raging in the city council meetings, with homeowner groups fighting against city councils who were elected on a platform of trying to get more apartments built.
I agree that job vs. housing density disparity is the cause of many of the problems. The way you fix the disparity is that you boost the housing density - you upzone near transit (SB 50 in the CA state legislature is trying to do this across the state), you increase property tax rates to discourage people from waiting for the land to appreciate without doing anything useful with it (ideally by gradually phasing out prop 13), and you streamline the redevelopment process by making it such that if you comply with a clear set of rules, you can start building with the knowledge that homeowners associations won't bog you down in endless lawsuits. And in my dreams, you change the zoning to mixed use commercial/residential so that you can colocate offices, restaurants, stores, and residential, making everything way more walkable and easing the burden on road infrastructure.
Paris: 27€/m^2 ($30/m^2)
SF: $4.95/ft^2 ($53/m^2)
Meanwhile, I understand your point that "many people are unwilling to live in small apartments" but those people are by default not part of the demand in San Francisco itself - they're part of the demand in Gilroy and Freemont, perhaps, but not San Francisco.
But "not being willing to live in small apartments" is not the same thing as not being willing to live in medium/large townhomes / row houses, which can accommodate vastly more people than live in most of the suburban expanse of the Bay Area. Just to make a dumbly over-simplified point, you could theoretically convert every ranch house in the bay area to two victorian townhomes and that would add room for more than a million new people, which I'm sure would make some dent in demand.
Likewise, most of San Francisco's land area is occupied by 1 and 2 story buildings (Sunset, Richmond, Excelsior, etc.). Just going to an average of 3 stories would add a hundred thousand or more new homes.
Regardless, while you may be right that demand is MUCH greater than supply and we'd need to add a million homes to make a dent, that's not a valid reason to do nothing.
If we're not going to build more housing here, why not limit employment growth here and create some pretty major programs to move jobs OUT of the city and take some of the housing pressure with it. There's nothing stopping us building a big new city at moderate density between San Francisco and Sacramento, for example, and if the whole corridor was linked by reliable, fast transit service then it doesn't need to be a traffic nightmare either.
TLDR; defeatism doesn't help, the economics here aren't THAT complicated we just don't seem to have the political will to do anything about it.
Most of the housing problems in western countries, from my point of view, seem to be that people want to live in places where they can't afford to live at.
I agree that housing is ridiculously regulated in many of these places, but I'm not sure that that's the only problem you have to deal with.
Even if you don't consider zoning laws, could we build enough supply to satisfy demand? People want to live in the city, but many are unwilling to live in (small) apartments. Many even want houses, but obviously the density of that type of housing is too low to satisfy demand. The city would sprawl. The outskirts would be far from where people want to live.