It helps to look at Hong Kong: similar to SF scarcity of land, excellent public transport, high population density and skyscrapers everywhere, yet it's even more expensive to rent than in San Francisco.
The way I see it: if living in San Francisco is 3 times "nicer" than living in location X, so the rent is 3 times higher. Changing zoning laws does not change the pricing, it simply changes how many people will be living in SF, but as long as it's "3 times nicer" than X, it will continue being 3x the price.
Hong Kong has about 7.2 million people in about 107 mi² of developable land[0]. In comparison, San Francisco has about 840,000 people in 47 mi² of land.
Additionally, San Francisco is just one piece of its urban area. There is a large amount of developed land outside of it that people can live in who can't afford the city proper. Hong Kong is much more self-contained, geographically and politically.
Furthermore, there is a vast amount of research showing that zoning laws hold tremendous (and quite possibly primary) impact over prices. To start, you might be interested in the book Zoning Rules! by William A. Fischel.
Excuse me for saying so, I can't help but think you made this comment with virtually no knowledge around zoning and urban development in general, and I don't want people to be misled.
[0] http://www.pland.gov.hk/pland_en/info_serv/statistic/landu.h...
No need to apologize, you're right: I am no expert. These kinds of responses is why I come to and comment on HN, thanks for the link!
The fix is really simply: combat the special interests and pass laws prohibiting the rent-seeking. Repeal zoning laws that are overly restrictive.
Harvard Economist Edward Glaeser discusses this in papers you can download. FT Columnist with BS, MS Economics Tim Harford discusses this in his book, "The Undercover Economist."
Not certain why people are proposing new solutions when the solution is already known. Helps if people study economics.