Not to defend facebook, but these kinds of indirect dependencies are only obvious in hindsight.

And even if someone thinks of the problem there's usually some kind of failsafe (an override key or angle grinder) in place.

Totally agree. A lot of time, money and effort can be spent to make everything redundant and it's one single point of failure you overlooked that will break the camel's back.

The point I think that we have to keep in mind here is that it seems that Facebook overlooked many many points of failure and was very sure of their own ability at doing everything and doing it perfectly all the time.

Anecdote from my previous life: Large corporation you would all recognize and which I won't name for obvious reasons. They ran their own data center(s) and it wasn't a side hustle to do that. One morning I come to the office and I have lots of messages about a massive outage. One of the data centers is offline. Like completely offline. I'm a bit fuzzy on some details because it's been a while but the main story went something like:

    External power goes out for whatever reason (don't remember)
    Batteries take over
    Diesel generators start up
    Batteries keep draining
    Diesels are spinning
    All Servers shut down

What happened to this massively redundant and well planned out data center? Nobody knew and again, fuzzy on the details but the report that came out a few days later was that somewhere in that system, was a non-redudant breaker/power line combo somewhere. And that breaker went flying for some reason.

This is why Netflix has a policy of routinely breaking things on purpose. If all your other practices are pretty decent, it's easy to build up false confidence based on all the disasters that you haven't actually prevented, just made unlikely.

One of the tools they have that helps with that:

https://netflix.github.io/chaosmonkey/

Facebook does in fact habe this as well, they regularly even take out entire datacenters to see if the remaining ones take over automatically.

So, Council Bluffs Iowa GOOGLE data center, disguised as the Gable Corporation. Got it.

Nobody expects a BGP issue. It's main weapon is surprise.

But, more seriously, nobody plans for BGP failures, BGP is a single system, with a distributed database across the internet, so there's no redundancy for it, and to add it up, it's nearly completely invisible, most people don't even know it's there.

I plan for a routing failure over my network, key points have a separate simpler OOB network on an independent internet handoff, with local hosts files and local username/passwords if tacacs is down.

This was not a DNS failure, nor was it caused by a single point of failure. Their entire backbone was taken offline due to a bad configuration change that went out to all of their routers.

Depends on how you define a single point. Single point doesn't mean 1 cause, it means at some point in the chain, there is a non-redundant component that when it fails the system stops. The fact FB could remove 100% of their capacity from the backbone is a SPoF if they don't have a redundant backbone completely separated from the configuration system used to control the primary.

I won't pretend to know the details of FB's infrastructure, and I'm sure there was a cascade of problems that made the outage harder to fix, but it sounds like years of separate teams building on top of other layers that were "guaranteed" to be online finally collapsed into a spectacular fireball.

How is a bad configuration change that took out their entire network NOT a single point of failure?

It's not like there were twenty different network engineers that typed the exact same command across their network. Ultimately it was one person who typed "enter" and submitted the bad change. And the problem wasn't even necessarily the change -- mistakes should be expected. The problem was the lack of a back-out strategy in case this happened.

How is that not a single point of failure? The failure was the one bad configuration.

You can say this about pretty much any outage or mistake ever. Every single security vulnerability or jailbreak too. How could the entire computing industry with 10000s of amazing people have missed Spectre/Meltdown? In 2018, Spectre affected basically every computer system ever - desktops & mobiles, AMD/Intel/ARM and it was basically open to exploitation since speculative execution which is decades. 1000s of employees is not as big as it sounds given how much FB does and hindsight is 20/20..