The root of Silicon Valley's successes and its problems are the same thing, and they both stem from how markets operate. New technologies in automation allow companies to successfully compete with incumbents but use drastically less manpower. (This, BTW, is why large swathes of America have seen mass unemployment and stagnant wages for a generation; you may not feel it in Lincoln, but the folks in Detroit/Ohio/West Virginia certainly do.) Markets price based on your best alternative, so if you invent a technology that does the same work as 10,000 people and price it accordingly, you can redirect the wages of those 10,000 people into your company's profits, and if you do that with 100 employees, you can all become pretty filthy rich.
All that money flowing into the Bay Area drives up prices, because there's a fixed amount of land (and some pretty fucked-up NIMBYist policies). The cost of living is an effect of the region's prosperity, not a cause, and not an independent variable. If you are a rainmaker for one of the companies - sales, or a key engineer or manager, or best of all, the owner of the company - then you're doing just fine; your salary goes up faster than rents & housing prices. But if you have to compete with all the other unskilled people in the area for an unskilled position, your salary is capped by what they are willing to work for; remember, in a market economy, you can price based on whatever the best alternative is, and there are a lot of good alternatives for chefs, housecleaners, and security guards.
The market allocates the small available amount of housing to the people who are willing to pay the most, which means that the folks who are not indispensable to this large-scale transfer of wealth from the rest of the world to Silicon Valley get priced out and either have to double up, move to Clear Lake or Stockton, or live on the streets. Most people from other parts of America are not willing to do this, so they move back home. However, these living conditions are typically an improvement on what you'd get in much of Southeast Asia or Mexico, so you see these unskilled jobs taken by them. (Many of the Indian & Chinese immigrants are actually very intelligent & highly skilled - U.S. immigration policy makes it very difficult to move from there if you're not. But when it comes to crowding & quality of life, the Bay Area is a literal breath of fresh air compared to China.)
The interest in UBI among Silicon Valley's cognoscenti is because they're looking ahead to what happens when the technology wave reaches the rest of America, and all the dentists, insurance providers, accountants, and other mid-level white collar workers have had their jobs automated away. What do you do with the people left behind? What do you do with the engineers once they've written all the software to automate our daily lives?
Most of Silicon Valley is actually fairly free-market libertarian, but because they're so exposed to the market and have first-hand experience with how it's changing, a certain segment here is thinking very actively about what the end-game looks like.
But haven't you read what the very empathetic and kind poster before suggested as an answer?: These fucking losers should move to whatever hill cave's still vacant, because they are to blame for not having gotten rich fast enough.
I really like your writing, very precise and to the point. However, you completely leave out the option of taxing and fair redistribution. Because, yes, I do believe everyone should have a right to live in the Bay Area. Maybe not in the richest neighbourhoods and beach houses, but there's got to be a fairer alternative to dividing up the space than pricing out everybody but the richest. Sure, there's the idea of UBI, but that still seems pretty far out and for most people is not a realistic option.
No amount of taxing and redistribution will achieve that, because the supply of housing in the Bay Area is restricted to grow slower than population growth. The only way to make that happen is to build more housing.
The solution is to build more housing and stop subsidizing one-another, so that those that can afford to live there do and those that cannot move.
https://en.wikipedia.org/wiki/Lincoln,_Nebraska#/media/File:...
You're an idiot.
As an interesting counter example, iirc one of the main groups of people who survived the Irish potato famine were the Celts, because they refused to give up the old ways of eating wild Clover and switch to growing potatoes.
Generally I agree though.
EDIT:
[0] http://www.health24.com/Natural/News/Can-humans-eat-grass-20...
Does that mean that those dying of famine could just go eat some wild clover, but they didn't?
Doesn't sound right.
During the famine, Ireland was exporting bumper crops of oats and wheat and other things. It was a completely avoidable tragedy.
Starving people would eat whatever they could to survive. There was limited aid that was privately funded and slowest to a trickle as tenants were unable to pay rents. 40% of the population died or emigrated, which reduced demand.
It's a situation the English have repeated in numerous countries and eras and one of the reasons I think England has a lot to answer for.
I have a feeling that you are seeing there a hyperlocal trend influenced by the particular local demographics. In the east bay from Oakland and north and in the mid peninsula, SF, and the North Bay, there are a lot of white families with kids (along with the other demographics you mention except perhaps fewer Indians)
can we stop perpetuating this myth!? Sure a lot of innovation has happened that area in the past but so much innovation happens elsewhere in the world. It's not the only tech Mecca and definitely not the best.
That's pretty much it. The Bay Area gives you the chance to be on the cutting edge of change, which - for a certain personality type that tends to be overrepresented in the Bay Area - is often reason enough to live there. Most of the people who want a stable & boring environment to raise a family have already left. Most of the rest are from regions where boring != stable. (Seriously: I'm expecting a kid in the near future, and in the childbirth classes, there are literally no white people - the only people having babies in Silicon Valley are seemingly Indian and Chinese, along with a smattering of Mexicans and other immigrants. I have a bunch of white friends here, and of the ones who have had kids or are planning to in the near future, they fit into two groups: 50% have moved back to their hometown already, and 50% are planning to.)
There's also the issue where history has basically never been kind to groups of people who felt they could live a stable, boring existence while clinging to old ways and enjoying a high standard of living. Makes me a little scared for America, but that reckoning is probably decades out.