1. I think that SF/SV's status as a "tech capital" is not fragile. Even if a dramatic push or pull happened it would be a slow drain. The dot-com crash was a quick flush, but that was predicated on the market realizing a majority of companies being revenue-negative wasn't sustainable. Google, Apple, Microsoft, and Facebook aren't going anywhere in the next few years. Despite newspapers reports of doom, most tech companies in the region are profitable. Tech employees' families are sticky and won't move unless the push or the pull is substantial enough to end friendships and uproot community ties.
2. Agreed, the SF/SV housing crisis has a large political aspect to it. I foresee it continuing indefinitely. Owners often immediately switch sides from YIMBY to NIMBY once they get their deed. The area is more geographically constrained than SoCal. Even if massive development were permitted, it would take half a decade before price reductions started to occur. It would be simple to plan this in a way to preserve the existing character of neighborhoods, but it would require centralized planning; only dramatic legislation at the state level would make this possible.
3. You should or you shouldn't live in SF. SF's culture has waned dramatically but Haight-Ashbury and Mission are still vibrant. I agree the SF counterculture is on the decline. Oakland has a lively art scene though it is also being choked by SF-like house price increases. I agree that it is at a turning point where many different things could happen.
I think that a rational person might still move to the area due to the feedback loop of successful companies congregating to where developers and capital firms exist, and developers moving to the region to get jobs at good companies. Though it doesn't feel like it to me, I can't help but admit it's a good thing because this price pressure will eventually push companies to open campuses in other metro areas. In the meantime, we can only hope and wait.
I wasn't necessarily arguing that SF will completely lose its "tech capital" title (though nothing's impossible), just that SF real estate is overvalued and that the craziness of this is likely to soften in the future as multiple factors conspire against it.
... though I do have to say: we are in uncharted territory. I have to reiterate that the Internet is young and is nowhere close to being fully assimilated by the culture. The net makes geographic diversification easier than it's ever been by orders of magnitude, and once more conservative industries adopt it this could have massively disruptive effects.
To give you an idea of how big such shifts can be: in the 19th century something like 90% of Americans worked in agriculture.
An effect of this magnitude could occur in relation to geography and place-advantage. The initial visionaries of the Internet predicted this, but they were overly optimistic about how long it would take. Personally I think they might turn out to be right but it'll take another 10-20 years to fully metastasize.
A maximum disruption scenario would be the "full virtualization" of centers. No more financial, tech, etc. "center" in physical 3d space. These things move to the cloud.
... and yes we are about to grow our team but our new team members are already identified. :)
I think the earth quake hazard is undervalued in comments asking for more construction, and at least when I lived there - I left about 10 years ago after having lived there for about 7 years - water was a serious future problem too. It was still okay but they were looking at huge amounts of deferred maintenance and construction just to update the water supply. Even when I lived there I could read about significant leakage of water for the peninsula on the way from Hetch Hetchy.
So if they decide to build more and more densely, I'm not sure what a) infrastructure like water will do, and b) what happens when the next major earth quake inevitably strikes - admittedly it is more likely to strike in the ("less valuable") East Bay.
Nor do I understand why it needs to happen. You said some of it yourself. There really is no reason for everyone to actually be there in order to work in an IT job. There are people who will argue that presence in their experience is a must, okay, so they can do that. But there are plenty of examples of projects and companies that have shown that purely remote does work for a lot of people and companies.
Why put ever more people into that small congested area that has to get basics like water from very far away, and will have huge problems with earth quakes? I immensely enjoyed living there, in places like the Presidio in SF itself, or in San Ramon, Mountain View and Redwood Shores (I kept moving) - and in the Oakland hills, a wonderful, gorgeous place with miles and miles of open spaces and trails and horses right in front of my door, with a view over the Bay to SF.
I don't see why the Bay Area of all places (meaning the physical limitations, water and quakes) should be made into Hong Kong density. Few people who (have to) work there actually benefit from the often touted benefit of "potential connections".
[0] https://ww2.kqed.org/science/2014/02/28/bay-area-do-you-know...
There also is the issue of how many people can actually be served from Hetch Hetchy (for the peninsula) and the various other places (various rivers) for the rest of the BA. Already Hetch Hetchy is being stretched. The water levels there also depend on the California drought situation. A headline from not that long ago: "Hetch Hetchy levels a major concern for water officials" (http://www.ktvu.com/news/4607391-story)
So adding a few hundred thousand or even more people to the Bay Area doesn't seem to be such an easy proposition and "purely political" to me? The politics sure also exist, but when I follow those discussions as an interested bystander - as someone who used to live there - I always hear the arguments about politics (which are of course true) but I never see things like water even mentioned. I'm not saying I know, I'm saying I wonder that it's not even mentioned. I would happily be educated and find out that those concerns have been solved or don't exist, but I'm suspicious because nobody even mentions them. Even if politics takes care of changing laws and starting construction, is it even feasible, especially the water question? And if I remember the climate model predictions right California will have more droughts than in the past decades (https://www.washingtonpost.com/news/energy-environment/wp/20...).
Predictions by a California government agency: http://www.water.ca.gov/climatechange/
Reading just the first few paragraphs on that page shows that they expect a negative impact on just the water sources the Bay Area depends on.
(1) How fragile is SF/SV's "tech capital" status?
SF/SV has no monopoly on talent. There are top engineers and devs everywhere. What it has a monopoly on is smart investors who are willing to take risks and top-notch accelerator programs.
It's ironic that the people who build the net can't use it to escape one of the most absurd rent bubbles in the Western world. The problem isn't that devs and founders aren't willing or able to do this. The problem is that investors and mentors tend to be older people who are not "net natives" and are not yet comfortable doing deals or mentoring remotely.
Younger angel investors -- or those who are more net-native and comfortable with telework -- will already fund things far away, sometimes without even physically meeting anyone involved. I see this happen regularly. Startup crowdfunding is also going to add a new source of geographically unconstrained capital.
As older VCs and older accelerator/mentor types who are not this comfortable with the net retire and are replaced, expect to see way more of this... especially since the real estate costs of SF/SV are a serious liability in terms of runway and recruiting capability.
(2) The SF/SV housing crisis is partly political.
SF/SV are geographically constrained and earthquake codes do impose additional costs on vertical construction. Those are indeed fundamentals. But the rest of the "shortage" is political and is due almost 100% to self-serving NIMBY resistance. That NIMBY base is being attacked from all sides now.
If the NIMBY wall crumbles, you'll see a construction boom to rival all construction booms. Supply will flood the market. If this also coincides with a dip in tech financing or factor #1 above coming into play, blood will be on the streets.
Edit: another factor is foreign speculation, and if SF/SV (wisely) copies Vancouver's new foreign real estate speculator tax that will help deflate this factor too.
(3) Why again should I come to SF?
Back before this madness SF was one of the greatest cultural centers of the world. It was quite honestly one of the coolest places in the world.
Real estate hyperinflation is pricing all that out. The artists and musicians and true 'hackers' and freaks and all the rest of the interesting people who made SF great can no longer afford to live there.
So why again should I pay through the nose to live in what's rapidly becoming a giant glorified office park for venture capitalists and big tech firms?
--
My own anecdote:
We've repeatedly resisted coming up there. I did the math. Doing so would triple our burn rate. We are "default alive" and growing today, but up North we'd be dead eight months ago. (We being this: https://www.zerotier.com/)
Investors who are not willing to invest outside SF/SV should just skip the middle-man and cut checks directly to real estate speculators.
The rent I would personally have to pay (I have kids) would be about 10% higher than my total cost of living in Southern California... and SoCal is not considered "cheap" by any standard. Last time I checked out apartments in SV I found a place that cost almost my total cost of living in SoCal where the lobbies smelled like dog piss. I call Silicon Valley "the world's only six figure slum."
The tech scene in SoCal is booming. Everyone I talk to says they have zero interest in going North because "you can't afford to live there unless you are rich."
Edit:
The parent is spot on, but I'd add this: while Manhattan is comparable to SF/SV, New York as a whole is considerably more affordable. That's because it has very good transit. You can easily live further away and commute into Manhattan, and you can read a book on your commute instead of staring at the butt of a car. You can also live without a car pretty easily in NYC and even surrounding suburbs, which really mitigates the cost of living. You can only do that in SF if you never want to leave the peninsula.
More and more people (and businesses) are wondering why they are allowing themselves to be raped by real estate when geographic diversification is easier today than it has ever been in all of human history. Seriously. Think about this. These real estate bubbles as we move into the true broad cultural uptake era of the Internet are like a bubble in fossil fuel prices six months after over-unity nuclear fusion has been demonstrated. Such bubbles have one and only one role: to allow smart money to exit and hand the (burning) bag (of dog crap) to dumb money. Every time you read about how this real estate bubble is permanent and will never deflate or how you are nobody if you don't live in one of 3 cities remember that you are probably reading a submarine ad sponsored by someone who wants to make sure they can exit at a healthy profit before reality ensues.
(Side note: given the Moore's Law growth of solar and batteries and the interesting progress in fusion I do wonder if the "peak oil" hysteria of the past 10 years wasn't exactly that... a final profit taking and exit opportunity for a soon to be obsolescent industry.)
In the long term SF/SV desperately needs a real estate crash if the region wants to have any hope of holding onto its culture or any of the other things that make it great. If a crash does not occur eventually the exodus from this madness will reach a tipping point and the city will lose its soul forever. Then the crash will still eventually come, but it'll be less like a correction and more like Detroit in the 80s.
Edit #2:
To avoid perpetuating age-ism I must point out that I know people in their 50s and 60s who are pretty "net native." It's a mentality thing. There is a bit of an age correlation but that's because not everyone remains open to letting their paradigm shift as they age. Ironically it is often the very successful who have big egos and inflated images of their own intelligence who resist this the most. It's one of the many ways that having a big ego makes you dumb.