Hurricane Electric? He, he, he.

It’s still batshit insane to me that Google Fi had any infrastructure overlap with Gmail. When I was there these grand unified base layer systems everything else was built on top of were a point of pride. From a risk perspective though it was sheer stupidity in retrospect.

As Google continues to atrophy and suffers attrition of the original people that built those systems, the probability creeps up more and more that someone will one day cause a catastrophic global outage of systems that should share absolutely no dependencies but do because “it scales”.

If Google made auto pilot for airplanes, it would all be on a centralized SaaS with white papers published in academic conferences about the elaborate systems designed to ensure server crashes don’t impact it. Nobody will ever complain about how all of those guarantees depend on chubby, big table, etc.

At my not Google job we talk about "what happens if a meteor hits a DC".

We agree that that is so rare that as long as there are buttons we can push to recover after a reasonable timeframe that is an acceptable risk, we don't need a fully automatic way to recover from that.

However your SRE teams needs a way to recover without intervention which is why there is talk of backups.

BTW even using different cloud providers isn't enough to avoid a DC outage necessarily. No amount of redundancy can protect you from it beyond a ton of services which intentionally slice off access to the DC leading to the risk of that happening accidentally which is its own risk.

But that’s incredibly stupid. A meteor hitting a DC is an intentionally dumb way to eclipse the much more likely risks of thousands of other things that can wipe out a DC.

I’ve been involved in such discussions and on the surface they seem reasonable but it turns into an easy reason to write of DC being wiped out.

In comparison there are far more likely reasons for a DC to be wiped out effectively permanently. Data centers at the base of WTC 1 and 2 are good examples. The myriad of targeted attacks on the power grid are also prime examples of attacks that would cripple a data center for weeks if they were in the cross chairs.

The Cascadia subduction zone has a much higher probability of wiping out all of them in the PNW simultaneously than a meteor hitting a single one.

Looking at it from a different perspective, there are other teams at Google who understand how sensitive their data centers are and they act appropriately. The list of data center locations is not public. A motivated group with long range rifles purchase from Walmart could wipe out a Google data center.

Google is very bad at modeling long tail events.

Meteor isn't real. It is literally just a metaphor and more fun than "we lost power" or "egress died". Anybody claiming it is impossible is silly.

You also won't lose a DC a year so you need some strict uptime guarantees to justify automatic roll over.

Prepare to avoid dataloss but needing SRE to change over is fine if you can stomach a hour of downtime.

> Meteor isn't real

Yes, just like "bus factor" isn't about buses

> The ultimate in "this would only apply at Google", was having to establish backup communication side channels that weren't Google dependent.

Not sure this is an "only at google" thing. In a past life, I ran Engineering for an App/SMS messaging application. We definitely used an IRC channel as well since the nature of our outages would mean messaging channels could be down. It's also why we didn't rely soley on SMS for our on-call alerting.

I think they meant it’s only at Google that Gmail would not be an uncorrelated email etc.

Though I think Microsoft probably has the equivalent problem with Exchange, Teams, etc.

I wouldn't say this is a Google only problem. Really, all of these problems of redundancy and recovery have been best considered by militaries well before private companies started to have sufficiently global operations and communications networks that they started thinking about it, too. In the Army Recon Course, they taught us to bounce signals off the ionosphere using hand-held radios if we needed to communicate with an otherwise cut-off remote unit too far past the earth's curvature horizon to get line of sight. And, of course, every unit is drilled in contingency plans to the point that it can operate independently as necessary even when all communication gets cut off and continue to push the overall mission forward even in the absence of further coordination, which, for what it's worth, obviously does present the potential for problems of its own when the mission changes and a disconnected unit doesn't know it, as well explored in Doctor Strangelove.