Except for bart, basically. And I certainly treat bart and MUNI as competitive products for commuting to work, although it doesn’t feel like MUNI does...
BART's service is long-train (up to 10 cars) through service on fully dedicated trackways following a single through route within San Francisco. If you happen to be transiting any points between Colma and Embarcadero strickly along that route, BART is your best bet.
MUNI Metro serves multiple endpoints throughout San Francisco, several lines of which transit the upper level of the Market Street subway. Outside that route, the metro shares rights of way with street traffic, resulting in drastically less predictable schedules. The requirement to manage tight-radius curves and street traffic also limits the length of metro trainsets to a maximum of two coupled trolleys.
The results aren't pretty, for MUNI. But that's all but a given under the environment they're operating in.
Not necessarily. They could exclusively run fast 4 car trains along the length of the subway, with transfers to street lines. That would prevent a single street-level train from backing up the whole system. http://newmunimetro.com/m-market/
You still end up with the service irregularities on the surface sections, unless those rights of way are dedicated. Something which really should have happened long ago on Geary and Irving/Judah, 3rd St., etc. Maybe with the dedicated transit route on Market the idea will spread.
Extending the subway further out is another option, though expensive and fairly unlikely.
With land based transit competition it is really hard to compete because land within walking distance is finite. (if you don't demand walking distance you get less riders by a lot). Thus to have real competition you need to have extremely high density - 50 story apartment buildings with few parks or other open space, at that point one line cannot handle all the riders and a second line can compete.
It's interesting to wonder why. I think one aspect is the degree to which politically powerful are willing to fight it. Famously Stalin actually cared that the Moscow metro worked well, and it did, in a country otherwise crippled by complacent behemoths.
Perhaps the Parisian elite likewise take more pride in having a metro which works OK, are willing to spend more political capital fighting for it. They feel some obligation to keep up with Brussels & Berlin, perhaps that substitutes for competition?
I have seen a lot of explanations of why the US pays so much more - they tend to not hold up. Note that last time congress asked the question the bill did not allow examining non-US construction for the report - something to write your congressman about.
Literally any amount spent on the subway will be returned in rent to landlords. You're just looking from the wrong perspective.
By who? And why should those that value it more than choice for the consumer value choice for the consumer more? And how would they even go about doing that?