The root cause of almost every San Franciscan problem is rampant NIMBYism. Look at a bus map on the city and notice how closely the stops are spaced together - you have buses that stop on literally every block due to local lobbying, and any attempt to remove some redundant stops to make things run faster elicits an incredible wail from local constituents for whom the only thing that matters is themselves.
Development is the same story. People moved to SF for the pretty Victorians they get to see when walking down the street, and any attempts to change this landscape, no matter how mindful of history, art, and architecture, is struck down to preserve the status quo (wherein the status quo is a sick, perverse emulation of the early 1900s).
San Francisco is the city where everyone is trying to squeeze through a very small door, while the people who just squeezed through are busy trying to slam the door shut behind themselves.
On a less bitter note: as someone who lives in the future (future circa 1930) and spends 0% of his day in unproductive commuting, you west coast folks are in for a treat in 20-25 years when (if) this happens. That extra hour or so a day of unstructured time to read, surf the net, or just veg is really wonderful.