The global outage drinking game:

bad DNS config push - 1 shot

routing loop - 2 shots

third party advertising your routes - 3 shots

power outage at data centre it turns out everything depends on despite decades and millions in engineering to avoid precisely that - 4 shots

Wolves ate through fiber - 5 shots

And it was a full moon - 6 shots

Single service failure, but service has not been restarted in 5 years, and no longer restarts in any documented fashion - 7 shots

And service developers left the company to found a startup - 8 shots

Expired internal SSL certificate - 9 shots

Daylight savings changeover-induced database corruption - 10 shots

Windows Update - 11 shots

5 is often broken down to “rain fade” and “backhoe fade.”

My favorite can’t-restart story is “the shared password manager is down, we need a hardware crypto key to restart it, the key is in the safe, and the safe’s combination was in the shared password manager.”

I was going to link a related comment but you're actually the author of it!

Here it is anyway: https://news.ycombinator.com/item?id=22316230

Coincidentally, I'm in the same predicament with Facebook, itself.

I can't log in to Facebook because Facebook wants approval from the Duo Mobile 2FA app. The 2FA app won't approve the login because it wants a code that can only be obtained by logging in to Facebook.

Solution: Deleted both Duo Mobile and Facebook apps. Moved on with my life.

isn't that like throwin the baby out with the bath water? i've never had problems with duo mobile, but then i don't use FB so i've never seen how it behaves with FB. so dump FB, but why dump duo mobile?

Because the only thing I used it for was to login to Facebook. All of my other 2FA is through other methods.

In a similar vein, the alternative option for 7 was a cyclical dependency introduced since last restart

I heard a story that Google SRE had to up a sacrificial Bigtable in any new zone before they could up enough services to finally up a real Bigtable for the zone.

Please, please tell me this is a true story. It's amazing.

I can only verify being unable to get the wi-fi password. Otherwise it’s all secondhand, I wasn’t on call.

> Wolves ate through fiber - 5 shots

I hope it's wolves, that would create a new category in https://cybersquirrel1.com/

You forgot:

Hunter shot your fiber relay with a hunting rifle - 6 shots

6 shots? It only needed one.

depends on the gun... some people have interesting ideas on hunting.

DST can be such a hassle. It affects our reports and change requests.

An adtech company once freaked out over a sudden 4% revenue drop, until someone pointed out that yesterday was 4% shorter than the day before.

Don't forget cosmic rays!

Apple decided to shut down Facebook.

Probably DNS. It's always DNS ;)

or BGP, or auth.

Large scale network partitioning can happen. Data center ingress/egress can go down. If you shard to multiple data centers, it's still possible for an entire autonomous system to have an interconnect with a different AS break. That is rare as hell with the actual Internet because of all the route redundancy even between ASes, but I've seen it happen on more specialized networks. For a few days last year, the entire western half of the US was cutoff from the eastern half on Raytheon's internal TS network because someone digging a hole somewhere accidentally cut the wrong fiber line.

DNS (not necessarily this case, but man it can bring things down fast)

How does it stay up as much as it does :)

"And if these people stop, the world burns. Most people don't even know what sysadmins do, but trust me, if they all took a lunch break at the same time they wouldn't make it to the deli before you ran out of bullets protecting your canned goods from roving bands of mutants."[1]

1. https://www.stilldrinking.org/programming-sucks

Blood, sweat, tears and caffeine.

also, some arcane shell script written by a retired engineer somewhere in the early start of the company.

I'll be curious about it coming back up. It's the sort of site where there's an army of people continually trying to reload the page if something isn't working.

Probably an intern doing something dumb.

If an intern can bring down the whole presence of your company online it's not the intern who is dumb.

While that's perfectly true, I'm sure a smart and determined intern could cause an astonishing amount of damage in many many orgs. Hopefully not at Facebook though.

The difference is whether or not it was intentional. If someone can bring everything down accidentally, the system is broken.

Ditto if they can bring it down intentionally and cover their tracks so no one finds out who did it.

some startups would like to disagree

bingo

The dumb thing would be giving the intern permission to bring down the whole site.

An intern shouldn't have those permissions.