The problems are twofold: geography, and the fact that SF doesn't operate in a vacuum. Geographically, there's water on 3 sides and hills on the 4th. No matter what you do SF is always going to engaged in an uphill fight against density because it's almost as much of an island as Manhattan is. The vacuum is the bigger problem. Even if the mayor and board of supervisors agreed to abolish the planning commission tomorrow and hand out permits to any proposal compliant with building safety codes, it would not do much to accelerate building.
Why? Because neighbors would sue under CEQA, the California Environmental Quality Act. This allows virtually anyone to file a suit alleging that a negative impact on the environment or an insufficiency in the preparation of an environmental impact report, anywhere in CA. This is not just 'the environment' in terms of wild salmon or old-growth forest, though - it includes everything from air quality to aesthetics. I'm expecting to be invited to join a suit on this because I live close to a hospital which is planning a large expansion, and the neighbors a few blocks away are furious about the likely change in traffic patterns and noise during the construction phase, which will last a few years. I don't actually care about this, but on the other hand I have misgivings about the proposed siting of a helipad right across the street from me.
And this leads to the observation that while CEQA has become a go-to example of obstructive red tape/environmentalism gone mad in California, it's actually just a culmination of much longer-standing legal traditions benefiting property owners. Any place that you have sufficient density for people and businesses to annoy each other, property owners end up in court with disputes about who should give way to whom. Grumble as one might about property taxes, nobody wants the value of their property to go down, either on paper in the more abstract sense of having less enjoyment due to (say) a reduction in the pleasantness of one's view or the amount of sunlight one's garden receives (which factors ultimately affect the $ value of their property anyway). If you have paid $$$ to buy a fancy loft with an amazing view of the SF skyline, you and your neighbors are going to be most unhappy if some developer proposes erecting a skyscraper and replacing your view with that of a concrete wall 30 feet away. In a more general context, people are much less enthusiastic about paying property taxes for parks and schools if the erection of tall buildings blocks sunlight from ever reaching them (and that particular complaint is a frequent one in SF, where the hilly geography tends to exacerbate the issue).
The complaints of property owners carry a great deal of weight in arguments over planning, since they provide a large part of the current tax base (vs the hypothetical future tax revenue of any given development) and angry taxpayers are highly likely to vote out officials who don't do their bidding and vote in those who are willing to pass laws restricting the use or development of property. Indeed, annoyed property owners have historically not just objected to new development but have often successfully argued for a halt to existing land use which impacts their enjoyment of property: in a famous English legal case, Sturges v. Bridgeman (1879), a doctor bought a house in a nice neighborhood and then successfully sued a nearby candy-maker, who had been there for years, over the noise of the candy-making. The court reasoned that while the candy maker had indeed been there first, the business was much noiiser than anything else in the neighborhood and it was only a matter of time before someone brought it up, regardless of the fact that the doctor had 'come to the nuisance.' That seems wacky, but think about it in the context of a city like SF that used to have a lot more light industry inside the city borders. If you know the city well at street level, I'm sure you've come across places where there's a bunch of shiny new apartments and hip restaurants, stores etc., except for that one wierd lot with a run down small manufactory/liquor store whatever, that either doesn't fit with the neighborhood any mroe or whose customers have gone away due to changes in traffic patterns. The business situated there makes less and less money, the owners maintain it less and less well, and eventually someone launches a blight lawsuit in an attempt to force the current occupant out.
The practical upshot of this for development is that a developer has to take into account the loss-of-value that a project will impose upon neighboring property as a potential development cost, and a wise developer will secure their agreement by offering to pay that cost to the neighbors up front, because otherwise they'll probably just end up paying the same amount in damages later (unless they gamble on winning an arms race of legal transaction costs, which is much more viable in rural than urban areas).
Good to see Krugman actually got something rightThen, he will almost certainly come out in favor.