http://idlewords.com/bt14.htm#america
"And speaking of California, let me try another tack.
This festive map shows seismic hazard in Northern California, where pretty much all the large Internet companies are based, along with a zillion startups. The ones that aren't here have their headquarters in an even deadlier zone up in Cascadia.
Now of course, each company has four zillion datacenters, backed up across the world. But how much will that matter when there's a major quake, and Silicon Valley can't get to work for a month? All of these headquarters are going to be shut down for a long time when the Big One comes. You're going to notice it.
So even if you don't agree with my politics, maybe you'll agree with my geology. Let's not build a vast, distributed global network only to put everything in one place!"
It's easy enough to find an off-season vacation destination and move everyone there; essentially anywhere is cheaper than SF, so that's easy. Not every company has to move to the same place.
It'd suck to be a service-industry worker in SFBA after an earthquake, but tech employers can just move, and hire service-industry people to fix up their offices for a few months before returning (or not).
(this might not be necessary after a 7, but after something insane like a 9, would definitely happen)
Depending on the situation, earthquakes can bring fires and tsunamis. So what do you tell the network engineer that had their house burn to the ground? "Sorry for your lose; we're decamping for Las Vegas. Wheels up in 8 hours"?
If the big one does hit there is a lot more at stake than just your precious data.
1. Immediately call extra hands in other locations (datacenters, customer service, etc); begin over-time scheduling to alleviate need for workers in disaster zone. Doesn't require any coordination of disaster zone, just remaining executives located outside.
2. Offer two special payments to workers in the impacted zone: 1. Disaster recovery assistance, to help them cushion emergency needs and get their family/belongings/etc recovered faster; 2. Relocation bonus, to create an incentive for quick relocation to temporary (or new) office.
3. Spend as much as you to have to get extra emergency services in to the area, and generally assist in government/regional recovery area. This will free up the last of your staff for temporary relocation.
I doubt that a large company (we're talking Google, Microsoft, Facebook, etc) would take more than a couple days to have other sites working overtime to cover the load, less than a week to have already transported some of their staff to a new site, and less than a month to have brought almost all of the office back online.
This timeline, of course, assumes that the problem wasn't that their office fell on all their employees, killing them or something of that nature.
Likely, we'd see a hiccup in the development cycle, but in terms of keeping infrastructure online, emergency bug fixes, etc, we'd probably see relatively little damage.
1. In most businesses these other locations are supplementary to HQ. Typically there is no one at the datacenter and CS centers rely on guidance from HQ. They are not in the position to make decisions and drive the company.
2. Growing up and working in Florida you natural disasters are like clockwork. I can say for sure that 1 is covered by insurance and the gov't and 2 is going to be hard to pull off without looking like a complete dick. Also, you have other constraints. Airports may be closed and driving can be dangerous. Plus cars need fuel which is typically in short supply and expensive.
3. If you have a company of a few thousand I would say a few hundred are essential. Where are you going to find a place to house a few hundred people at once? Remember that other companies are thinking the same as well.
I also believe the most essential employees are that way for a reason. If they have the same preparedness in the company they also have the same mentality in their community. It would not be illogical to imagine the person in charge of DR for the company is also a member of the volunteer fire department.
Microsoft would be able to recover, then they find out the caterers don't have food to serve the cafeteria, employees can't come to work because there is no fuel, etc. These are the soft issues that most DR plans don't account for.
The end result was a high reliance on contract employees at the fail-overs, and a real appreciation for the human element.
But then again, I keep forgetting that the 'real' valley is just for early 20s-fresh-out-of-school-startup-single-guy-whizz-kids apparently.
If all your data is in your own building, you might be okay (assuming you can get at it, stuff it into a truck and move it out). Otherwise you're going to be waiting for a while.
There might be a run on laptops, monitors, chairs, etc which overwhelm the Hawaii and Las Vegas retailers for a while, but basically California's high energy costs, low power circuit availability, high sales/use tax on servers, etc have solved this problem already.
SFBA is critical for personnel, not so much manufacturing or hosting.
There is about 3mm square feet of datacenter space here. There are very few large companies which have servers only here and not somewhere outside the area but which would be reasonably expected to continue operating if their servers were somehow unaffected.
However, SFBA isn't even the first location for most companies I see. They go either into the cheaper AWS cloud regions (us east or the Oregon), or managed hosting somewhere (rarely SFBA).
Bay Area companies which directly get colo early on are fairly rare: stripe, square, etc. I fully support it as a strategy, but it is statistically insignificant.
Aside from price, east coast single location also gives you a lot better latency to Europe. Asia is usually screwed anyway, but the extra 80ms makes a big difference.
Well, putting everything in the Australian outback doesn't help that much, because there are good reasons why thousands of tech workers can't work there. This is essentially an argument against having any population centers in earthquake zones.
Which is an interesting point in and of itself.
http://www.nytimes.com/interactive/2011/05/01/weekinreview/0...
In the list of worst predictable natural disasters in the Continental US, it rates one above the expected earthquakes to the south, the result of two plates mostly sliding past each other: https://en.wikipedia.org/wiki/San_Andreas_fault
Worse, in part because we aren't at all prepared for it (aside from I hope more serious buildings) would be a return of major earthquakes in mid-West: https://en.wikipedia.org/wiki/New_Madrid_Seismic_Zone (something I pay attention to since that's on the other side of Missouri from me).
I forget one of them, and the worst, which few in the US would survive, and where being in Australia might not be a bad idea at all, would be a return of the Yellowstone Supervolcano: https://en.wikipedia.org/wiki/Yellowstone_Caldera
If the whole of the Americas was wiped off the Internet tomorrow then, for those of us in The Rest of The World (TM), life would go on. I think the only web property that would be annoying to not have would be Wikipedia. Second to that is the Google search engine but I take it as a given that they could cope just fine if America just mysteriously vanished one day.
I think that there are some that think that Silicon Valley is the centre of the universe, some indispensable magic place where all of the innovation happens. If there was The Big One (for the valley) we would soon find out exactly how important things like social networks and iphone apps really are. Life would go on. If anything, a big earthquake in Caliphonea is far preferable to a big eathquake in Japan. In Japan they make stuff the world needs.
If the whole of the Americas was wiped off the Internet
tomorrow then, for those of us in The Rest of The World
(TM), life would go on.
????What about the DNS roots, all the sites running on AWS, any site using Cloudflare or the JQuery/code.google.com CDN, etc
>DNS roots
http://www.iana.org/domains/root/servers
>AWS
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-reg...
>Cloudflare
https://www.cloudflare.com/network-map
>code.google.com CDN
https://developers.google.com/speed/libraries/devguide#Intro... - no location list though, only "globally available"
What we'll notice is not having filters for the pictures of our Starbucks cups. The demographic of teenage girls in malls will be devastated.
Silicon Valley is no longer the epicenter of technology; it's the epicenter of the fashion of technology.
It sounds like you're making an over-generalization based on the last few years of "social" while overlooking many of the largest tech companies in the world. Apple, Oracle, HP, Cisco, Google, Intel, the list goes on, are all headquartered here. http://www.siliconvalley.com/SV150/ci_25548370/
It was a poor place to make the comment; it wasn't on topic. My apologies. But when the comment was made that zillions of startups would be wiped out, and that we would notice, I wanted to clarify that we might notice but only in the most superficial way possible. Yes, it's a generalization -- there are always exceptions, and there are large tech companies there, but there are large companies in every major metropolitan area. If HP can't brand laptops for a year, I think we'll somehow manage to all survive.
I made the comment because I wanted to correct what I saw as an antiquated notion: that SV was different from Houston or Singapore. It used to be. It's not anymore.
If you want to look at devastating datacenter loss, if anything happened to Northern NJ the markets would go into a tailspin.
I'm just tired of the obviously incorrect proposition that Bay Area is producing just frivolous photo sharing and social communication tools (which both has a huge value for humanity, but that's a different story) and I seriously find it an interesting thought experiment what would happen on a global scale if the Bay Area would be devastated by a big earthquake.
The obvious tech giants like Google, Apple, Oracle and Cisco are arguably supporting a huge number of businesses and organizations around the world. But a glance through a list of just publicly traded US SaaS companies reveals that about 40% are from the Bay Area. These offer SaaS-based solutions ranging from HR tools (Workday), IT service monitoring (Splunk), ITMS (ServiceNow), CRMs (SalesForce), ERPs (NetSuite), Insurance backends (Guidewire) to supply chain management (e2open), to just name a few.
If Bay Area suffered anything resembling 2008 Sichuan earthquake, it would halt a good part of the economy for a while.
For one, insensitivity. There could be some victims even today, and you're talking casually about an event that will wipe out the Valley and S.F.
Second, most data centers are in tons of places for redundancy. Google, Microsoft, Amazon, etc etc. Even low cost providers like Linode And Digital Cloud have redundant data centers in 3-4 locations around the wolds.
Third, ever done a ping to Australia? When most of your users are in the US, it's not really a good idea to have data travel halfway around the world and back. Even if anything else improves, the speed of light will remain constant.
Fourth, you think Valley based datacenters are not built to endure earthquakes? That they are just some dumb houses or something with racks inside?
I don't see why tragedy should stop us from thinking rationally about the situation.
>Second, most data centers are in tons of places for redundancy. Google, Microsoft, Amazon, etc etc. Even low cost providers like Linode And Digital Cloud have redundant data centers in 3-4 locations around the wolds.
The problem is that something between 1/3 to 1/2 of all connections to Asia pass through California[1].
>Third, ever done a ping to Australia? When most of your users are in the US, it's not really a good idea to have data travel halfway around the world and back. Even if anything else improves, the speed of light will remain constant.
The machine I have in Sydney pings 60ms to MIT on average. Caltech, UWS and UCLA are all down from there which probably has something to do with the Earthquake.
>Fourth, you think Valley based datacenters are not built to endure earthquakes? That they are just some dumb houses or something with racks inside?
Are the cables made of magic unicorn hair that doesn't break? Because if [2] or [3] happens I don't care how much like a bunker your data center is, it's not talking to anyone. Which also leads back to the problems in number 2, namely, far too much bandwidth passes through California for how dangerous it is.
[1] https://www.technocrates.org/wp-content/uploads/2012/10/subm...
[2] http://www.geerassociation.org/GEER_Post%20EQ%20Reports/Ital...
[3] http://www.cloacina.org/blog/wp-content/uploads/2012/03/crac...
Ping measures round trip time, not point to point.
Distance from Sydney NSW to Boston, MA: 16,230 km
http://en.wikipedia.org/wiki/Optical_fiber: "From this information, a simple rule of thumb is that a signal using optical fiber for communication will travel at around 200,000 kilometers per second. "
Some basic math: (16 230 km) / (200 000 (kilometers per second)) = 81.15 milliseconds
that's one way. You should be seeing no bnetter than 160ms, and even that is idealized because there is switching equipment that induces delays.
PING e9566.b.akamaiedge.net (184.24.249.86) 56(84) bytes of data.
....^C
--- e9566.b.akamaiedge.net ping statistics ---
728 packets transmitted, 724 received, 0% packet loss, time 10177ms
rtt min/avg/max/mdev = 25.935/38.167/131.406/16.421 ms, pipe 11, ipg/ewma 13.999/35.015 ms
http://www.geoiptool.com/en/?IP=184.24.249.86
My the server I'm testing this on is in Sydney, the server info on the geoip says I'm pinging somewhere in Cambridge, Massachusetts.
In particular, your network path is sending you to a server which is in australia.
$ host 184.24.249.86 86.249.24.184.in-addr.arpa domain name pointer a184-24-249-86.deploy.static.akamaitechnologies.com.
trimmed mtr output:
19. sun1-ge5-0.gw.optusnet.com.au 57.1% 8 186.4 184.9 182.1 186.4 2.3 20. 198.142.144.70 42.9% 8 186.9 182.6 180.3 186.9 2.9 21. sun1-ge12-0-0-500.gw.optusnet.com.au 0.0% 7 179.7 180.3 179.5 182.3 0.8 a184-24-249-86.deploy.static.akamaitechnologies.com 22. 198.142.144.70 0.0% 7 185.6 184.6 179.5 194.2 4.9 a184-24-249-86.deploy.static.akamaitechnologies.com 23. a184-24-249-86.deploy.static.akamaitechnologies.com 0.0% 7 185.1 184.0 178.8 187.8 2.8
that's a cache of MIT's web site hosted in Australia (Akamai edge cache)
http://research.microsoft.com/en-us/um/people/ratul/akamai.h...
geoip is worthless in many situations, this being one of them.
file /bin/ping
and you will see that it is setuid to root.I don't see why thinking rationally would entail talking without any sensitivity and humanity -- which is what I complained about, not the "thinking rationally" part itself.
>The machine I have in Sydney pings 60ms to MIT on average.
Doesn't 60ms sound quite bad as the optimal average one can have?
>Are the cables made of magic unicorn hair that doesn't break?
No, just that buildings are make of scientific structural construction that endures and absorbs earthquake pressure.
As for the cables, that's an easier problem to solve, and it will be more localized.
"A repeat of the 1906 magnitude 7.9 earthquake, the worst case scenario for the Bay Area, is estimated to result in about 5800 fatalities if it strikes during working hours. Most scenarios, however, have maximum projected fatalities on the order of several hundred, reflecting the success of earthquake-resistant design and construction practices in California, particularly in residences. The loss of life is predicted to be highest if an earthquake occurs in the early afternoon when people are working in commercial buildings with varying vulnerability to quakes These predicted mid-afternoon fatalities are generally about 5 times higher than values predicted at 2:00 AM when the population is assumed to be in wood frame residential units."
Also - there are a ton of data centers out in Nevada, Oregon, DC, etc...
http://www.usa.com/nevada-state-natural-disasters-extremes.h...
http://www.usa.com/washington-state-natural-disasters-extrem...
http://www.usa.com/washington-dc-natural-disasters-extremes....
It will be literally cheaper and faster to send homing pigeons with hard drives around their necks than to have to wait for the same amount of data to be downloaded [3] and [4] will become a viable way of implementing ip.
Probably the first industry to be hit by this new disruption will be music or books when every song/book in the world could be stored on a phone. For example the spotify library at 320kbps would take up something like 250 Terabytes, which will be the size of a laptop hard drive in 8 years give or take.
The upside to this is that movie studios will probably push 16k screens and extremely high frame rates to keep files big enough that you can't just have one drive with a few thousand movies on it.
[1] http://www.nngroup.com/articles/law-of-bandwidth/
[2] https://en.wikipedia.org/wiki/Mark_Kryder
[3] http://www.smh.com.au/technology/technology-news/carrier-pig...
You can easily store all of Hacker News locally, at our computer, so why do you come here? For news, obviously. You can't have news pre-saved at your computer by definition, and you simply won't accept (nobody does) the few-hours ping time of the sneakernet.
By the way, carrying media around was always faster and cheaper than delivering the data through the Internet. The trend we are getting is exactly the opposite of what you described; more people are choosing the net option, because although it gets relatively more expensive all the time, in absolute terms it's getting good enough.
Of course, things like the power grid and water systems where there are no competing providers, those are more likely to leave people completely without service.
Even the incentives to be robust or even safe are greatly reduced. A company with a legally protected monopoly can even blow up half the houses on a street (as PG&E did in San Bruno) and all the rebuilt houses are guaranteed customers.
There's 34 listed just for San Jose and Santa Clara...
Earthquakes don't destroy digital infrastructure. At most they might dislodge a few things temporarily.
Historically you might note that fires have sometimes occurred as a result of earthquakes, but then it's unlikely those fires were in data centers.
Stuff does sway around a bit, but it's not a huge deal for the digital infrastructure.
I think the real risk here is the loss of life (bus factor).
Here are some useful maps:
http://www.hazardscaucus.org/presidentialdisasters65-98.JPG
http://www.datafoundry.com/wp-content/uploads/2011/04/natura...
Also, if it helps you sleep better at night, none of the root dns servers are in the bay area: http://public-root.com/root-server-locations.htm
FWIW, I would tend to agree - in general terms.
True, though there are a lot of people working in that area. Deaths at the wrong time could be catastrophic.
Also, while a lot of companies are in the SV, I don't think many servers are actually there. Google wouldn't disappear from an earthquake.
An earthquake is not an endgame either. Sendai and Kobe are still on the map. Though I don't know how prepared SF is for this.
Why not build it in some frozen area instead? All the same advantages and your cooling costs are ridiculously cheaper.
But don't put anything close to the seed bank, which a crazy with a nuke (IS, Putin, Iranian ayatollahs, etc) might want to target...
Edit: Maybe not Alaska for geological stability.
(I did not downvote, btw)
Ultimately the coast of California is getting set up to be the biggest case of hubris and short-term thinking in human history: Where we knew for decades what the significant risks were, with ever increasing accuracy and sureness, but kept going ahead anyways. If such a major event eventually happens, as every seismic model predicts, I don't think people will say "Well, we knew this was coming.", but instead will point fingers, question why the government didn't prohibit building up there, etc.
Put it in the middle of the Australian outback, nothing's happened there for a billion years, nearly literally.
To the people down voting me: please explain why you think it's a good idea to put our digital infrastructure in a place which might well be destroyed in an afternoon when it can be put quite literally anywhere.