Re:Employees given three months to return to Facebook office
by jaywalk on Thursday October 07, 2021 @12:22PM
(#28788432)
This was not a DNS failure, nor was it caused by a single point of failure. Their entire backbone was taken offline due to a bad configuration change that went out to all of their routers.
[ Parent
| on HN ]
Re:Employees given three months to return to Facebook office
by treesknees on Thursday October 07, 2021 @12:53PM
(#28788956)
Depends on how you define a single point. Single point doesn't mean 1 cause, it means at some point in the chain, there is a non-redundant component that when it fails the system stops. The fact FB could remove 100% of their capacity from the backbone is a SPoF if they don't have a redundant backbone completely separated from the configuration system used to control the primary.
I won't pretend to know the details of FB's infrastructure, and I'm sure there was a cascade of problems that made the outage harder to fix, but it sounds like years of separate teams building on top of other layers that were "guaranteed" to be online finally collapsed into a spectacular fireball.
Re:Employees given three months to return to Facebook office
by mbreese on Thursday October 07, 2021 @12:57PM
(#28789031)
How is a bad configuration change that took out their entire network NOT a single point of failure?
It's not like there were twenty different network engineers that typed the exact same command across their network. Ultimately it was one person who typed "enter" and submitted the bad change. And the problem wasn't even necessarily the change -- mistakes should be expected. The problem was the lack of a back-out strategy in case this happened.