This is a multi-faceted problem and must be addressed hand in hand with transportation. What if we could house everyone but nobody could get anywhere? Higher density housing along transit lines, better public transportation, etc. are all things we should be working to improve. But we also need to consider the demand side of the equation. Is there a way to even out the demand. What if we build a ton of new high-density housing and the economy takes a major downturn or the tech industry gets distributed to other locales? Will we be left with a bunch of empty buildings that will be a drag on our regional economy and downtown areas for decades?
I feel the biggest immediate relief to the demand issue is legislating tax breaks for companies who allow some large percentage of their workforce to work full-time remote. This would immediately allow people to live in lower cost areas but still take advantage of the dynamic Bay Area tech economy. This is the quickest way to release a little pressure on both housing and transportation while working on the longer term solutions.
* You're completely right. Transportation needs fixing. This is mostly a governance problem, I think. There are too many agencies, too much overhead, and too little accountability.
* Demand management always struck me as needlessly authoritarian. Why do it so explicitly, when prices are doing it for us? And what keeps politicos from going "Well, we convinced a bunch of the evil rich techies to leave. Now they can be replaced by worthy natives! Mission complete!". It would be dangerously politically popular, and lots of local politics tend towards populist and short-term-ist.
Have you looked at how much of SF and the Bay works in tech? Most days, it feels like the answer is absolutely everyone. Yet, in SF the actual number is something like 6% of the workforce. That's half of the number working in finance. If a sixth of the tech workers left SF tomorrow, the population would be replaced within a year or two (recent rates are around 1% yearly). The approach might get some very short-term relief, but without major reform a lot more time would be needed to get long-term changes in place. Never mind the politics of handing tax cuts to tech companies!
Your monoculture concerns are real, valid, and rooted in reasonable worries. You are absolutely right to have them! Yet, they could be better-aligned with reality.
I really wish these two things were bound together.
With that said, prop 13 has wreaked havoc on budgets all across California. School funding is a state-wide problem. And I would go so far as to say that the whole of the Bay Area experiences a housing shortage for what is mostly the same set of reasons.
So while you're right, there may be some commonalities to be found.
However, I think it might actually be possible to build homes rapidly enough to meet demand. We should at least try, and if we fail we'll have a lot of new housing. That's a nice prize no matter the outcome. What's convenient about this approach is that if we get rid of the political obstacles to construction, we almost certainly can build enough housing!
Failing that, how about we tax people who have helped perpetuate this state of affairs and benefit the most to help address the problem? How about a "created this mess" tax on everyone who has been in the Bay for ten years or more? Maybe they can contribute a little to solving the problem they created, rather than inflicting pain on other people and shouting "STOP HITTING YOURSELF!".
No, that's not a serious proposal. It makes more sense than slapping people with taxes for the privilege of moving to one of the most expensive places in the world, but that is not a high bar.