As soon as you start digging underground you find a whole litany of old pipes, wires, mains, etc that do not appear on any drawing. You're unsure if they're in use or not, and some of them are so old that just pulling the earth away from them causes them to fail.
There's more to it other than politics. These things are hard to do in established cities, especially at the shallow depths that muni operates at.
More context: https://www.citylab.com/transportation/2018/01/why-its-so-ex...
At some point we need to understand the trade offs that come with regulations and either redesign them to enable large scale infrastructure projects or accept the fact that such projects are just not possible in the current framework.
So for eg environmental regulations to deter coal plants are fantastic because they encourage the alternatives. But regulations that prohibit anything seem as useless as no regulations: we need to generate power somehow.
Seems like corruption is potentially the only big differentiating factor here.
Another plausible factor is the push to privatization and contracting in the US public sector removing effective management from the picture, so that the public sector is no longer capable of cost control. For example, by realizing when bidders on contracts are bullshitting too high or too low, or by pushing people to suggest cheaper designs.
A corollary to the previous point is that the US generally seems unwilling to look at foreign countries (especially countries not named Canada or the UK) for insights into what can be done better. This can result in people not adopting newer, cheaper technologies in a kind of NIH syndrome.
Another possible factor is that, for several kinds of projects, the US's diverging rules (e.g., FRA regulations basically banning all European rolling stock until they were revised a year or two ago) combined with a relatively low install base could be preventing economies of scale to make things cheaper.
There's lots of plausible factors. Actually digging into what makes them so expensive is very time-consuming, and I suspect there is very little pressure (let alone ability!) from anyone on any side of the political spectrum to actually make it happen. It's much easier to pick the factor you personally believe in and say it's so without any evidence to back it up.
This seems to be a big factor also in companies. My company’s IT department has outsourced a lot and so a lot of projects are managed by people who can’t judge if something is good or not. I have heard the same about government agencies. They have no experts left.
AFAIK, examples are: 1) The east bay bridge ramp repair after that tanker fire 2) The Culver CIty <-> Santa Monica portion of the LA Expo Line
AFAIK, both of these finished early and the first one "under budget".
An oil truck tipped over, lit on fire, melted the steel in the overpasses (same level, above, and below). Bids were put out; you got a bonus if you got it done in under a month, and then for every day before that, further bonus. One bid came in both super low on "cost", and in time (like a week or something), and they'd make all their money on the bonus structure. Firm succeeded (lost like a day to a flat tire on a shipment, too), and AFAIK they were the firm selected for the Yerba Buena changeover. No quality issues either, although I don't know details.
For LA...
LA expo line was originally not going to reach the beach until... nowish? But something happened, and it got done years early. AFAIK, that schedule increase didn't come with a cost increase. Literally right now, my understanding is that the Beverly Hills line is getting accelerated, since there's no traffic to interrupt / work around.
Edit:
Has any government project been on time and overbudget, or are they always both over time and over budget?
The resulting built thing is a distant afterthought.
My personal hypothesis is that Americans get very defensive about this because there's a cultural resistance to being the guy who got fooled, so many of us will invent reasons that fall down really fast like "SF is an old city with lots of underground things." or "SF is in an earthquake zone" or (my new favourite) "Liquefaction", etc. etc.
My suspicion is that SF these days is more like an oil emirate. It can pump all this free tech money, and doesn't need to worry about having a functioning society around it.
Just an unfinished thought.
San Francisco's subway extension.
https://en.wikipedia.org/wiki/Central_Subway
3 years behind schedule, with work starting in 2012 and a cost of 1.6 billion dollars for 1.7 miles of rail.
The same story repeats itself all over the country. https://www.nytimes.com/2017/12/28/nyregion/new-york-subway-... New York clocking in at 2.7 billion per mile for a 6 billion dollar extension.