I think this is what SF locals like to invoke as a reason. I don't really think it bears out. Paris is full neighborhoods, but it's a dense city. Tokyo is dense with thousands of small neighborhoods. Taipei is dense with hundreds of small neighborhoods with their own character. all these cities realise they are urban and they need to accommodate growing populations. Basically, it's red-herring.
>...but building a residential high-rise in the FiDi? Look at the Transbay Terminal project
SoMa is the only place where it's somewhat less onerous to build. But the FinDis? Not quite, even if it was historically amenable to tall buildings. Last time a tower was attempted, it was rejected on the basis of "character" and casting shadows[1]. I really dislike the "character" argument. It's as if things were not meant to grow and change. It's as if they'd want the city to be a "living" museum. It's a romantic, but unnatural notion, to me. I want some progress, sure, manage it, so it does not end up like Mumbai, or something like that.
>any time you're talking SF, there is the elephant in the room...
Lots of cities live on fault lines. LA, Seattle, Tokyo, Taipei, MDF., etc. You need to build them to code. And, not only that, but we already have tall buildings, it's not as if this is Phoenix, so this is beside the point.
> Bay Area is better than most in that regard
I would disagree with that. The Bay Area has 6 million+ residents. It's one of the most poorly coordinated regions with such a large population.
W/re views. I realize that's how it is, but I think it should be more a public good. How does NYC, Tokyo, MDF, Paris and so on deal with this? Obviously it's not disallowing growth in those cities. As long as they don't build monstrosities like they do in Caracas, I think sensible blending is okay and if you lose some view, so be it and be happy you had something which wasn't actually yours. I mean, SF thinks of itself as "progressive" but then, something as vain as "view" is like a sacred cow. how does that make sense?
[1]. http://www.sfgate.com/bayarea/nevius/article/Tower-new-front...
Tokyo has about the same density as San Francisco (6,000/km^2), though maintained over a significantly larger area. The main thing that's interesting about Tokyo is that it's medium-density but very large: http://marketurbanism.com/2012/06/28/tokyos-surprising-lack-...
> How does NYC, Tokyo, MDF, Paris and so on deal with this?
Paris has taken a split approach, where high-rises are banned in the city center, but a new high-rise-building space was set aside in the suburbs. When Tour Montparnasse [1] was built in 1972, it was so unpopular that further such buildings in the center were banned. New high-rise construction is therefore moved to a few km out of the city center, to La Défense and the suburbs. An analogy might be allowing high-rise construction in Oakland or Daly City, but not in SF proper. The mid-rise buildings are higher-density in Paris than SF, though, typically 5-8 stories throughout the city center.
Or else, cultural differences? In much of Europe, block-sized, mid-rise apartment buildings, often centered on courtyards, are seen as a very classic, nice sort of city-center construction, while high-rises are seen as a bit garish, either too much like an office building, or else too much like an ex-Soviet housing project. Americans seem to have the opposite association, of high-rises as modern and mid-rise apartment blocks as meh.
(Though on the ex-Soviet angle, I've read that as one explanation for why Berlin is one of the cheapest major cities in Europe: the East German government didn't have much patience for these bourgeois concerns about "historical character" and "neighborhoods", so just built rows of high-rise apartment blocks right in the city center. As a result, the city now has an unusually large housing stock.)
On the economic side, I think cities prefer single family devs because they can levy more taxes (as they do it per acre, I think). I don't see why they don't tax by livable (inhabitable) space and land, that way they can compensate for the greater density in required services.
Anyway, I've begun reading Jane Jacobs's "The Death and Life of Great American Cities"[1]. It provides interesting insight into the suburbanization of America (and other places, of course, but she looks at the American side of it) and its impact on quality of life and how it may be time to rethink the suburbanization and revitalize cities to make the inner cores actual livable places, not just places where people drive in for work. It's an old tome, but it's still very relevant today.
W:re Soviet style (and even section 8 in the US and Council Estates in the UK) the planners were kind of utilitarian and naive. They more or less saw them as solutions for housing people as if they were storing objects. Little in the way of allowances for human nature. The plans were a bit too abstract. For example, let's say they thought it'd be nice to have a community area. They'd afford a room somewhere in a building, but put little thought on where might be optimum or how people might use it and have equally easy access to it. So they might end up being dingy unused areas which might just get defaced or used by low level criminals.
[1] http://www.amazon.com/Death-Life-Great-American-Cities/dp/06...
In general, I don't actually see a limited supply as a bad thing. As long as office space and residential stay within the right balance, the mix should be sustainable. The goal should not be to make every city look like Manhattan. The suburb I grew up in effectively ran out of land, demand increased in the next town out, and it got built up. There is no real shortage of land in CA as a whole - It doesn't take much of a road trip from SF to find an area where cows outnumber people, and it's reasonable that groups of people are able to define the character of an area in which they live.
The slippery slope view argument doesn't really hold water, largely because much of the city was built a century ago, long before the current residents settled there. Zoning restrictions caused views to be priced into the value of the property. Current owners bought their properties largely with the view-restrictions intact. You buy something, you know what the view you have, you know what view you don't have, you know what restrictions there are on what you can do with your property, and it's priced accordingly. It's a reasonable system for all.
I do second your suggestion of better regional planning as a whole, though, mostly with regards to transportation, though the Bay Area is better than most in that regard.