Wait, what? There is a reason things are the way they are. People do not want you to live there, affordably. Period. Cities are "designed" to be expensive and exclusive. That, in part is their value.
Whilst I've never seen San Francisco for myself, it sounds like they are trying to buck the trend and go for expensive and exclusive (as you say), with various complicated planning and zoning laws. It is definitely against the norm however.
I questioned your premise that cities were "designed" to be expensive, whereas I believe they merely "become" expensive through basic laws of supply and demand. The reason cities such as London and San Francisco never grow ever taller skyscrapers is because of planning laws and regulations that restrict supply - not, a free market of supply and demand.
London is a good example of such in that it has ancient viewing rights that restrict the height of nearby buildings, the classic case being that no building can be higher than St. Pauls (a relaxed principle nowadays, but still enforced sometimes). This is a major reason why the old houses in London haven't been knocked down to be replaced by a whole skyline of skyscrapers, but instead are massively expensive properties compared to the rest of the country.
Capital cities are a "premium" product, and they are designed to be so. People pay the premium, because they are effective in the task for which they are designed. Where do you think the "demand" comes from? The demand is for the productivity improvements that can accrue to those who control assets that are highly productive. But note the two words: (1) control; and (2) productive. To extract maximum rents, the product must be under control (so expect a class of laws designed to g;tee this). And secondly, expect a design which caters to high-end productivity (ie, networking, and its corralry...social exclusivity).
If you need to brush up on the history of new york and san francisco as 'financial capitals'.
SF may be a financial capital to some degree, but your comment about VC is still off: they're not based in the city, and they didn't take much interest in companies in the city at all until recently.
You can find decent housing in NYC, with pricing comparable to other smaller metro areas in the U.S. like Seattle or D.C. without too much fuss. Same goes for Paris. That's why most people in those cities don't live in the center. Brooklyn or Queens, for example, has a larger population than Manhattan. But only Staten Island has a lower population Density than SF.
The problem with SF is that it has ~Manhattan prices without the number of available units. This means that laughably high housing prices get pushed further and further away from SF proper and out into the Bay Area.
Arguably, if you think of the Bay Area as one city, this makes sense, it's the same in any major city.
It's not so much that someplace like Cupertino or Santa Clara are expensive by themselves, but that you're paying much more for similar housing than you would a similar distance from other similar sized city centers. Yet people move there and saddle themselves with ridiculous mortgages because the alternative is to live in someplace like Stockton.
Dramatically increasing the density of SF proper could make the outer areas and even the East Bay area much more reasonable places to live. However, it would mean lots of people who bought high in the current market would lose their shirts if such a thing were to happen.
I remember the very first time I visited SF, my friend was regaling us with tales of the insane housing prices. No slouch and from a pretty expensive area myself I figured it was just an issue of demand like you'd see in any other big city. But when I actually got to the city I was floored with how low the density obviously was. The solution to the high housing prices seemed idiotically simple then as it does now. Build SF up up up! Areas around every major BART station should also be higher density areas. Relax the density further out from these centers and you'd end up with a much more livable city plan like pretty much every single other city in the country.
By way of comparison, the D.C. metro area (not exactly a well planned urban paradise) has about the same population density as SF Bay Area yet housing can be had for probably around 2/3s the price at comparable distances from the city center.
And there's a greater diversity of opportunity here if you're not just in the tech startup scene. I have a feeling SF will bust when the VC spigot dries up.
[1] https://en.wikipedia.org/wiki/San_Jose-San_Francisco-Oakland...
[2]https://en.wikipedia.org/wiki/Baltimore%E2%80%93Washington_m...
the city remained a walkable, urban paradise compared to most of America.
Most higher density cities are even more walkable urban paradises. Plus they have the added benefit of far better public transport than SF offers.
SF: 17620/sq mi
NYC: 27550/sq mi
Paris: 55000/sq mi