Rationally speaking, you're 100% right. Those are all true impediments. From a realpolitik perspective, it makes total sense to start building where it's not controversial so that parallel non-engineering work can be undertaken. There are rational reasons why labor is so expensive, why planning committees take so much time, and why things take as much time as they do.

But this isn't how large public works can get done quickly. Any sufficiently large project is less technocratic and more political. Allowing large projects to be governed entirely technocratically has shown that timelines stretch so far out as to render the project impossibly expensive to finance and too limited in scope by project's end. There needs to be a single authority (with checks on it...) that can get things done, governed by a politician who can ensure that the authority's efforts are directed to popular projects. An agency with a planning body that makes community-approved blueprints ready before funding comes in, and with the authority to exercise eminent domain on approved plans and start work quickly and immediately.

So while it's cheaper out in the Valley than in LA or SF, it's still significantly more expensive than a non-grade-separated, say, ~75-100 mph-rated track would entail, the sorts of which are common in Europe and do a decent job moving people around cities spaced at such distances.