My intuition suggests that the rents being so damn high is yet another case of the 1%, already rich, screwing the rest of us to maintain their cash flow. However, as the late John McCarthy cautioned, "he who refuses to do arithmetic is doomed to talk nonsense" [1]. So, I'll need to do my arithmetic before jumping to this conclusion. Once I'm finished, I'll post an "Ask HN" question on this issue.
[1] Progress and its Sustainability, by John McCarthy
Location still matters, or else there would be no such thing as startup hubs like silicon valley. I'm thinking more along the lines of...why is that outside of downtown SFO, most buildings are at most 2 stories high? Is it just due to prevailing zoning laws and rent controls or is there something more? There is obviously great demand for housing as indicated by the extremely high rents in this area, so why not just replicate the downtown high rises further south?
By the way this is what I found on a plaque next to the Golden Gate. The biggest faction opposed to building the bridge were the landlords in SFO, who feared that once the bridge was built, it would greatly lower the rents they could collect.
Relax regulations, clean out zoning, remove market controls, employ people on the black market, do the construction not-for-profit and you might be able to slash the cost by half or maybe two thirds, probably not more. (There is a finite amount of people willing to pay to live in such a high rise, but let's assume demand really is huge.)
In particular, even at that lower construction cost it won't be worth it for anyone to build if the rent is to be comparable to maintenance costs, whether through supply/demand or otherwise. This will be the case until you can put up high rises at a total cost of about $900-1200 per unit (roughly 15-20 times the monthly payment), no matter how many you can put up.
[1] http://persquaremile.com/2011/01/18/if-the-worlds-population....
[2] http://shine.yahoo.com/green/simple-life-in-manhattan-a-90-s....
http://financiallyfit.yahoo.com/finance/article-113345-10728....
[3] http://www.baycitizen.org/columns/elizabeth-lesly-stevens/sm....