News for hackers. Stuff that matters.
Re:The Really Big One by Locke1689 on Monday July 13, 2015 @01:45PM (#9879690)
Hmm, in the spirit of thinking of worst-case scenarios, most of the engineering talent for Amazon and Microsoft is in Seattle and Redmond, respectively. There's even a sizable percentage of Google cloud platform in Fremont and Kirkland.

Even if most of the employees live, it's likely basic infrastructure will be a disaster for quite a while. Maybe Google would be able to offload ops work to Mountain View, but it seems likely Amazon.com, AWS and Azure would go down, possibly for months. I wonder what the effect would be on the global economy.

Re:The Really Big One by alxv on Monday July 13, 2015 @02:47PM (#9880125)
Google runs a company-wide disaster recovery testing event every year. Earthquakes are among the numerous scenarios simulated.

https://queue.acm.org/detail.cfm?id=2371516

Re:The Really Big One by sneak on Monday July 13, 2015 @06:45PM (#9881664)
Re:The Really Big One by idlewords on Monday July 13, 2015 @03:08PM (#9880297)
The problem is that you can't really simulate the systemic effects of a really major quake (like a scenario where Mountain View is out of power for weeks).
Re:The Really Big One by nostrademons on Monday July 13, 2015 @03:22PM (#9880403)
The DiRT drills include the simultaneous loss of whole offices, including the complete obliteration of the Bay Area. Participants are instructed to pretend that they've been eaten by zombies and not communicate with anyone in a remote office, and that's backed up (as described in the article) by cutting the network links to & from the Mountain View corporate office.

Now, there may certainly be larger systemic effects over long time periods caused by eg. the death of the company CEO, the loss of new product development, or the complete destruction of the company's market (in, for example, a global thermonuclear apocalypse that sets us back to the stone age). But the tests absolutely do simulate the simultaneous death of 15,000+ employees, destruction of the Mountain View headquarters, and loss of all datacenters, data, and power in the Bay Area, and verify that remote teams can continue the day-to-day operations in such an event.

Re:The Really Big One by Sven7 on Monday July 13, 2015 @03:55PM (#9880602)
What happens if Google goes down for a while? Is it really that bad? Yes, Gmail and Drive may cause all kinds of chaos but I think I can survive quite a while without the search engine and YouTube.
Re:The Really Big One by rryan on Monday July 13, 2015 @04:26PM (#9880811)
The data could be lost. Part of DiRT is ensuring that the result of losing N data centers is not "oops we lost 20% of GMail users data permanently".

On the scale of devastation this article is describing, none of Google's servicing being obliterated really matters but I'm personally pretty glad Google is prepared for multiple, massive scale disasters to happen at once.

Re:The Really Big One by baudehlo on Monday July 13, 2015 @07:26PM (#9881833)
The financial disaster would be horrendous. Look at the number of companies using google for email.

When the towers went down in 2001, the first business priority of many companies was getting email back online. It's an incredibly important part of a business.

Re:The Really Big One by trentnelson on Tuesday July 14, 2015 @09:04PM (#9889126)
Reminds me of when I was doing some consultancy for large oil/energy trading company in London. There was huge DR initiative going on at the time, and one thing they realized after surveying the business is that, if worst came to worst, they could still conduct business if they lost every other business system, as long as e-mail was available.
Re:The Really Big One by elorant on Monday July 13, 2015 @04:40PM (#9880903)
If Google App Engine goes down many sites will stop working. OAuth services will also start to fail. So will custom search provided to 3rd party sites (ok that may not sound too bad but you never now). Then there is cloud storage which in many cases is used as a backup mechanism. And how about the gazillion of sites that load js libraries from Google's CDN? Sure, a day or two is bearable, but make it a week and then the shit hits the fan.

The one thing you should bear in mind about earthquakes is that in many cases the aftershocks make more damage than the main one, because damage accumulates. Which means that critical infrastructure could take weeks to become fully operational.

Re:The Really Big One by pixl97 on Monday July 13, 2015 @04:32PM (#9880854)
>I think I can survive quite a while without the search engine

The search engine that holds the vast majority of the worlds traffic? The search engine that everyone is going to call on minutes after the earthquake to get news, but isn't going to be there so all the other search engines slow to a crawl. If Google search just disappeared for a week there would be a considerable number of businesses go out of business. You see it now in one off cases where a site is delisted and its traffic drops 99%.

That said, Google has datacenters around the nation and the world so that scenario is unlikely.

Re:The Really Big One by TrevorJ on Tuesday July 14, 2015 @12:39AM (#9882923)
I wonder how it would play out if all the major search engines went down. I think people would start communicating on whatever sites they knew the URL for that were still up. Probably some ad-hoc manually curated indexes would pop up in comment threads on various sites and then a few people would probably aggregate these and host them on their local boxes.
Re:The Really Big One by rodgerd on Monday July 13, 2015 @05:12PM (#9881131)
> Yes, Gmail and Drive may cause all kinds of chaos but I think I can survive quite a while without the search engine and YouTube.

Perhaps you can, but how many companies are now wholly dependent on Google for their groupware functionality? What happens if you're one of the gaggle of people who make a living off YouTube?

Re:The Really Big One by pc86 on Monday July 13, 2015 @04:06PM (#9880687)
In the grand scheme of things, not a lot of businesses use Apps for their email, but I'd hazard a guess that you're still looking at hundreds of millions of dollars in losses if that were to suddenly disappear.
Re:The Really Big One by Mtinie on Monday July 13, 2015 @05:43PM (#9881351)
It's the cascading service failures and related economic costs that would be the most damaging (and hardest to really quantify until a scenario really happens).
Re:The Really Big One by sneak on Monday July 13, 2015 @10:16PM (#9882461)
As with most people who reason about Google, you are off by at least two orders of magnitude in your estimate.
Re:The Really Big One by gaadd33 on Tuesday July 14, 2015 @11:57AM (#9885865)
Wow, I had no idea that Google Apps were about 15% of Google's revenue. That is a way bigger than I would have guessed. (Hundreds of millions x 2 orders of magnitude is >= 10B, total revenue in 2014, 66B)
Re:The Really Big One by pc86 on Tuesday July 14, 2015 @01:16PM (#9886465)
I'm not referring to Google revenue, I'm referring to productivity/revenue for the businesses using Apps day to day.
Re:The Really Big One by Locke1689 on Monday July 13, 2015 @05:24PM (#9881220)
Maybe Google is distributed enough to stay up, but I'm pretty sure Microsoft and Amazon are not, and I doubt Google has the spare capacity to handle Azure load, much less Amazon load.
Re:The Really Big One by miles932 on Monday July 13, 2015 @06:45PM (#9881668)
Try us :)
Re:The Really Big One by Locke1689 on Monday July 13, 2015 @08:12PM (#9882038)
If Google utilization is really only 10% of capacity, that's pretty surprising and worrying for a host of other reasons.
Re:The Really Big One by sneak on Monday July 13, 2015 @10:14PM (#9882455)
Utilization and availability are not directly related. Much of their utilization is probably not super time critical and could be proportionately shed or suspended in an emergency situation like the one in question.

I bet Google could absorb 100% of the emergency evacuation of AWS in a pinch. Remember also that not 100% of services will be moved, and those that are will not be running at 100% capacity - everyone would be suspending their batch/analytics/warehousing jobs that week (or month).

EDIT: Also, lock in would stop all the redshift/dynamo/etc users from migrating, and would limit people to just what they could throw up on GCE/GAE.

Re:The Really Big One by yokohama11 on Tuesday July 14, 2015 @01:25PM (#9886529)
Most large companies do/have done exactly that. Arguably 9/11 was the big catalyst for DR planning.

There were Wall St firms that literally had to get the military to escort them into Lower Manhattan in the days after to rescue servers, tapes, etc and cobble something together in a satellite office.

No one wants a repeat of that, and you'll notice subsequent disasters like Sandy have barely been a blip for most as a result of policy changes.

At this point basically any major company will/should have procedures in place for "perfect" DR, the ability for everyone to work from home indefinitely (and alternate offices/infrastructure waiting for the few people who need something special, like the special needs of a trading desk), as well as obviously having back up data centers with full replication.

Re:The Really Big One by falcolas on Monday July 13, 2015 @03:21PM (#9880391)
Sure you can. Just look for the keys to the main breakers to those buildings. Of course, paying for that amount of vacation for 15 days is probably not entirely feasible for any company.
Re:The Really Big One by idlewords on Monday July 13, 2015 @05:30PM (#9881252)
By 'systemic' I mean not confined to one company.
Re:The Really Big One by kraig on Monday July 13, 2015 @06:01PM (#9881451)
tsunami monkey?
Re:The Really Big One by PotatoEngineer on Monday July 13, 2015 @08:30PM (#9882117)
The Microsoft campus in Redmond would most likely not be knocked over by the tsunami: a rather hilly Seattle is in the way, then there's the large Lake Washington, and then Redmond is another ~40 feet above the lake on the far side, plus another ~2 miles inland. A fair amount of the energy of the tsunami is going to be expended by the time the Microsoft campus gets salted. (Kirkland is west of Redmond, adjacent to Lake Washington, so it's a similar-but-somewhat-less-optimistic story there. Fremont is adjacent to Seattle and slightly inland, so it will be worse there.)

Of course, the infrastructure of the greater Seattle area is going to be trashed, but chances are good that the Redmond and Kirkland campuses of Microsoft and Google respectively would be intact.

Link to the inundation zone predicted for a magnitude 7.3 earthquake on Seattle: http://wa-dnr.s3.amazonaws.com/Publications/ger_ofr2003-14_t... This isn't even "the big one" as the OP describes, much less "the really big one", but it's a rough description of where the flooding will happen. Downtown Seattle (up and left of the word SEATTLE on the map, with streets in a dense diagonal grid) won't be strongly affected, because downtown Seattle is a hill. With an 8.5 or 9.0 earthquake, of course, Seattle will be much worse off, but there's still some hope that Redmond and Kirkland won't get too damaged.

Re:The Really Big One by waynecochran on Monday July 13, 2015 @04:25PM (#9880804)
Not only would there be significant downtime of major internet services, but lots of people would die as well!
Re:The Really Big One by qq66 on Monday July 13, 2015 @04:33PM (#9880857)
Downtime of major internet services could also cause deaths elsewhere, as more industries become reliant on cloud services.

I could imagine hospitals losing access to health records, emergency responders losing access to mapping or route planning software, people losing access to Uber and not being able to get to the hospital because the taxi companies have gone out of business.

Of course all this stuff is supposed to have clean failover, but not all of it has had its mettle tested.

Re:The Really Big One by hdunlop on Monday July 13, 2015 @07:17PM (#9881788)
like Uber, but for once-in-an-epoch natural disasters
Re:The Really Big One by zecho on Monday July 13, 2015 @05:06PM (#9881096)
Man. In times like these—the northwest falling into the ocean or whatever—it's unimaginable to think about the sheer amount of data loss that could occur. I'd probably be really worried about that loss of data or, worse, the loss of services if millions of people suddenly died.
Re:The Really Big One by myth_buster on Monday July 13, 2015 @06:07PM (#9881485)
Isn't it the case that there is replication across data centers which are geologically spread apart?
Re:The Really Big One by deegles on Monday July 13, 2015 @06:21PM (#9881549)
Only if you had the foresight to set that up.
Re:The Really Big One by Locke1689 on Monday July 13, 2015 @05:19PM (#9881183)
Well, I live here, so, yeah. I think the people who live here understand we're screwed, the point is what the larger effect will be.
Re:The Really Big One by markonthewall on Monday July 13, 2015 @05:26PM (#9881231)
I believe the parent comment was referring to the numerous hospitals or other critical services who rely on these infrastructures.
Re:The Really Big One by cko on Monday July 13, 2015 @09:33PM (#9882325)
Re:The Really Big One by jodrellblank on Monday July 13, 2015 @05:53PM (#9881407)
It's almost like you didn't read the article.
Re:The Really Big One by waynecochran on Monday July 13, 2015 @06:03PM (#9881462)
I read the whole thing in gripping detail with Mt St Helens out my office window... I was imagining the horror of not being able to blog about my death as the Tsunami was approaching...
Re:The Really Big One by jodrellblank on Tuesday July 14, 2015 @12:12AM (#9882845)
Then you'll realise the human impact was mentioned repeatedly in the article - population in the affected regions, elderly people, school children, estimated death counts by different organizations. That was one main focus of the article - how many people such an event would kill.

Why is it necessary to prefix/suffix a comment on the article, with an "of course, people will die and that's obviously bad, we don't want people to die, is this enough mentioning of the people dying that we can say something else without you passive aggressively implying I don't care about the people dying?"

Re:The Really Big One by waynecochran on Tuesday July 14, 2015 @11:57AM (#9885867)
Just seemed like horrific lack of perspective. The infrastructure supporting transient internet technologies is way way down the list of importance if the Cascadia Subduction Zone slips.

I grew up in Central Oregon which is surrounded by fresh (geologically speaking) lava fields. I wonder if the Native Americans that were around when Mt Mazama was turned into Crater Lake worried about the disruption of their Obsidian tool making business.

Re:The Really Big One by prawn on Monday July 13, 2015 @07:42PM (#9881894)
I think the human toll goes without saying (well, until you said it). The technical impact is naturally of interest specifically to HN readers.
Re:The Really Big One by platz on Monday July 13, 2015 @11:54PM (#9882795)
Especially since we know HN readers are sociopaths
Re:The Really Big One by TrevorJ on Tuesday July 14, 2015 @12:44AM (#9882944)
Shh, nobody is supposed to know that.
Re:The Really Big One by JPKab on Monday July 13, 2015 @05:11PM (#9881122)
That would be absolutely horrific, a loss that I can't even comprehend.

The loss of life would be sad too. /s

Re:The Really Big One by jsprogrammer on Monday July 13, 2015 @02:18PM (#9879916)
I believe you are giving reasons why our computational infrastructure needs to be further distributed. We should not be dependent on a small number of companies who wish to locate their resources in continental subduction zones.
Re:The Really Big One by serve_yay on Monday July 13, 2015 @01:53PM (#9879754)
That does seem likely, though as far as I know, AWS's biggest zone is located in Northern Virginia. And, I'm sure the AWS people have planned as much as they can for this eventuality. (Um, I hope.)
Re:The Really Big One by Locke1689 on Monday July 13, 2015 @02:18PM (#9879917)
I doubt the datacenter's locations matter. The services at the bottom of the stack need constant ops to stay alive. With most of the engineers out, I expect barely a few days to go by before one of the essential services needs servicing.
Re:The Really Big One by torinmr on Monday July 13, 2015 @03:17PM (#9880364)
For companies that size, the really important services probably have a distributed ops team, with a team in Europe/Asia alternating 12 hour oncall shifts with a team in North America. In an emergency, the non-North American team could take 24 hour shifts to keep the infrastructure alive.

Though of course if headquarters was really wiped off the map, there'd be huge problems with communication, tribal knowledge that only existed in the heads of senior engineers in Seattle, and so forth.

Re:The Really Big One by jsolson on Monday July 13, 2015 @09:16PM (#9882274)
When supporting a service on the scale of say, EC2 and the rest of AWS, one would typically expect follow-the-sun support.
Re:The Really Big One by stephengillie on Monday July 13, 2015 @01:55PM (#9879766)
AWS West in Oregon are somewhat further inland from I-5. It would probably be impacted, but not to the same degree as the datacenters in other cities like Tukwila.
Re:The Really Big One by sneak on Monday July 13, 2015 @10:17PM (#9882470)
The loss of datacenters is irrelevant; the loss of people to keep the services running is the issue.
Re:The Really Big One by trhway on Monday July 13, 2015 @04:29PM (#9880831)
i think the datacenters will get soon to the status of national strategic infrastructure objects similar to bridges, power plants, etc... Given the distributed nature of the services i'd suppose the main threat though isn't from physical effect like earthquake, instead i think the main threat is introduction of something like StuxNet into say AWS (or highjacking the backdoor access that services like NSA already have there)
Re:The Really Big One by kelnos on Tuesday July 14, 2015 @09:20AM (#9884617)
Sure, the AWS us-west datacenters would go down (Oregon and NorCal), but us-east (VA) is still the largest of all AWS regions. While Amazon is headquartered on the west coast, I sincerely doubt a failure of even both us-west regions would take out the others.

Ultimately it's up to the user: if they're going to put all their eggs in a us-west basket, they're out of luck if the west coast ends up under water.

Re:The Really Big One by hinkley on Monday July 13, 2015 @08:09PM (#9882027)
People talk and talk about how The Cloud is about decentralizing things but really at the end of the day it's about centralizing everything with about a dozen vendors. And of course when price sensitivity causes attrition that will turn into 4 or 5 vendors.

I'm not sure I see how that really counts as 'more robust'.

Re:The Really Big One by chrismcb on Tuesday July 14, 2015 @02:01AM (#9883125)
There is an entire mountain range between Seattle and the ocean. I am curious how the Sound would affect the tsunami.