Totally agree. A lot of time, money and effort can be spent to make everything redundant and it's one single point of failure you overlooked that will break the camel's back.

The point I think that we have to keep in mind here is that it seems that Facebook overlooked many many points of failure and was very sure of their own ability at doing everything and doing it perfectly all the time.

Anecdote from my previous life: Large corporation you would all recognize and which I won't name for obvious reasons. They ran their own data center(s) and it wasn't a side hustle to do that. One morning I come to the office and I have lots of messages about a massive outage. One of the data centers is offline. Like completely offline. I'm a bit fuzzy on some details because it's been a while but the main story went something like:

    External power goes out for whatever reason (don't remember)
    Batteries take over
    Diesel generators start up
    Batteries keep draining
    Diesels are spinning
    All Servers shut down

What happened to this massively redundant and well planned out data center? Nobody knew and again, fuzzy on the details but the report that came out a few days later was that somewhere in that system, was a non-redudant breaker/power line combo somewhere. And that breaker went flying for some reason.

This is why Netflix has a policy of routinely breaking things on purpose. If all your other practices are pretty decent, it's easy to build up false confidence based on all the disasters that you haven't actually prevented, just made unlikely.

One of the tools they have that helps with that:

https://netflix.github.io/chaosmonkey/

Facebook does in fact habe this as well, they regularly even take out entire datacenters to see if the remaining ones take over automatically.

So, Council Bluffs Iowa GOOGLE data center, disguised as the Gable Corporation. Got it.