So, yes, SF is a small space but it is far from being efficiently (or fully) developed and I think the SF Planning Department and other bureaucracies are partially to blame.
[1] http://missionlocal.org/2012/07/el-chico-produce-store-to-op...
For example, the reason that space at 18th and Mission was vacant is right there in the article you linked -- the landlord pushed out the previous tenant, and turning it into a grocery store required extensive renovation. It's not as if the space was sitting empty because nobody had a use for it:
"So when the 99-cent store closed — because of high rent, according to neighboring businesses — and the vacant building went on the market in May 2010, he seized the opportunity and bought it two months later."
But yeah, it would have been nice if the new owner allowed someone else use the space while waiting for his redevelopment plans to go through, instead of letting squatters turn it into a drug house.
[1]http://www.auto-parker.com/productDetail.aspx?Id=93&Name=PUZ...
That said, it doesn't change my point: to change the density of these places, you have to buy out a bunch of people who know exactly what their land is worth if it's zoned for higher density. Many of them are actually betting on that windfall. There is no equivalent in SF to the vast tracts of undeveloped land in, say, Texas.
And how to fix it? It takes a long time and a lot of investment to improve public transport, but it's challenging when the need for it is driven by short-term boom-bust tech "bubble" cycles. Makes it harder for the city to invest when demand might go poof in a year.
It takes decades as an artifact of governmental processes. There's no necessary reason it should take decades.
That kind of thing is good for preserving historic buildings too - the blocks are all in nondescript areas 1-4 miles from the center.
Paris: density 55,673/mi^2
SF: density 17,246/mi^2
(source: wikipedia)
All of NYC - 28,053/mi^2 Manhattan alone - 72,033/mi^2 London - 13,410/mi^2
12 SFH homes on a block can be purchased for $30 million, have 400 unit buildings built on them and make far more than that selling it off. Then take the massive increase in property tax revenue and use it to fund an expanded rail system with right of way. But they can't do it effectively because of the law.
Build 250 of those 400 unit buildings and you will have filled the 100'000 unit backlog. 250. You can easily fit them all 0.5 miles from bart and muni tunnel stations. Everyone knows the solution, the planning department and SF NIMBYs don't want to happen, and they fight it hard.
250 high rises wont even make SF approach hong kong or manhattan levels. Not even 1000 of them. SF and it's people has it's head so far up it's ass it's comical.
The last time this city was on discount was probably the late 1980s, when SOMA was a wasteland and everyone wanted to live in the suburbs. And surprise surprise...along came the first dot-com boom. All of those (comparatively) cheap spaces where creative kids were having raves and making art became the birthplace of the modern web. Creativity happens in marginal spaces.
I was actually surprised that pg kept emphasizing the need for constraints on development in this essay ("The empirical evidence suggests you cannot be too strict about historic preservation. The tougher cities are about it, the better they seem to do.") The conventional wisdom around here is that we must refer to this kind of thinking as "NIMBYism", and condemn it as backwards-thinking. "Obviously," says the technorati, "we should be knocking those tiny old buildings down and replacing them with skyscrapers!"
But he's right. San Francisco was once a cheap, pleasant place to live, precisely because of those sorts of constraints, coupled with demographic trends. But it's at a different point in the development curve today. It isn't for scrappy, poor innovators anymore. It's for ambitious, get-rich-quick types who will tolerate paying $4,000 a month for an apartment to be part of the scene. And as soon as the scene moves somewhere else, those people will be gone, too.