It's true that the Bay Area as a whole needs to get denser, but SF proper needs to lead the way.
I guess another phrase this thesis is that SF's apparant NIMBYism is a misleading artifact of what you describe as a "historical accident of municipal lines."
This distortion, of course, can make SF look good as well as bad. You can pump SF up by talking about the ratio of <desirableculturalfactor>:Resident compared to Houston, or Chicago, or LA, conveniently ignoring the fact that in Chicago, the residential suburbs are counted, whereas in SF, they aren't. My guess is that this ratio in LA or Chicago's densest 48 square miles would look quite similar to SF. Or, alternatively, you can make SF look bad by excluding the sort of developments that favor the burbs (such as percentage of households with children - when a family moves from SF to Redwood City, SF "loses" a family - in many areas, this would just be considered a move within the region).
Good or bad, my thesis is that these differences are largely an artifact of how the borders are drawn, rather than innate cultural differences between SF and other metropolitan areas. I don't doubt that these cultural difference exist (and they may in fact be driven by the fact that the electorate in SF isn't diluted by the votes in the surrounding areas), but I think they may be greatly overstated.
http://en.wikipedia.org/wiki/San_francisco http://en.wikipedia.org/wiki/Los_angeles
The Bay Area as a whole is less dense than the LA Metropolitan area.