It could be otherwise than it is, that's sort of the truth of network effects, but it's difficult.
And then you get the chicken-and-egg problem: with all those workers there, companies want to locate there to have access to that talent pool. And then the workers want to stay there so they have access to all those jobs that they can hop between.
Living in SOMA requires $120k/yr minimum to responsibly afford renting a 1 bdrm. Getting paid 20% less but, having housing costing 1/4 of what it is in SF is a win overall. And this is coming from someone who loves the tech scene in SF. So many smart people, so many great ideas, so much energy. But, to move there, I'd have to make at least $35k/yr more just to break even and that's living in a hovel in comparison to where I live now.
Even putting aside the ethical and moral arguments, which carry a lot of weight for me: (1) There are network effects in urban environments. (2) Globally, specific industries often concentrate in specific cities. A lot of tech has concentrated in the Valley up to the San Francisco. The effects are getting stronger as that migration occurs. That's a great thing: it allows for people to move between companies easily, allows ideas to be shared, etc, etc.[1] That effects is and will continue to be hard to replicate elsewhere. (3) Destroying that seems like a gigantic waste. (4) We are destroying it through our current public policies -- housing and other. (5) I believe there are political solutions, just not ones that have been well-organized.
[1] Urbanism has a lot of other benefits too, if done right: like less traffic, and a lower per capita carbon footprint.