Some more:
- Google 2016, configuration management bug/BGP: https://status.cloud.google.com/incident/compute/16007
- Valve 2015: https://www.thousandeyes.com/blog/steam-outage-monitor-data-...
- Cloudflare 2013: https://blog.cloudflare.com/todays-outage-post-mortem-82515/
Sounds like the same happened here:
"Due to this withdrawal, Cloudflare engineers experienced added difficulty in reaching the affected locations to revert the problematic change. We have backup procedures for handling such an event and used them to take control of the affected locations."
But Cloudflare had sufficient backup connectivity to fix it. I'm curious how Cloudflare does that today-- the solution long ago was always a modem on an auxiliary port.
Also lets face it - the utility of a trusted security guard/staff with an old fashioned physical key is pretty hard to screw up!
I'm surprised more places don't implement a "click here to confirm changes or it'll be rolled back in 5 minutes" like all those monitor settings dialogues
Now you can use mobile Internet (4G/5G)
I'm not blaming BGP, since it prevents far more outages than it causes, but BGP-based outages have been a thing since its beginning. And any other protocol would have outages too - BGP just happens to be the protocol being used.
It just seems that most of these are local enough and the Internet resilient enough that they don't cause global issues. Maybe the exception would be AWS us-east-1 outages :-)
BGP is just a tool, it would be something else to do the same purpose.
The main difference between BGP and all other tools is that if you mess up BGP, you've done a very visible thing because BGP underpins how we get to each other's networks. But it's not a sign of BGP being fragile, just very important.
Another canonical example is C++. Some tools make it easy to blow your leg off. Some tools provide safety mechanisms to stop the saw from cutting off your finger.
- Cloudflare 2022 (this one)
- Facebook 2021: https://news.ycombinator.com/item?id=28752131 - this one probably had the single biggest impact, since engineers got locked out of their systems, which made the fixing part look like a sci-fi movie
- (Indirectly caused by BGP: Cloudflare 2020: https://blog.cloudflare.com/cloudflare-outage-on-july-17-202...)
- Google Cloud 2020: https://www.theregister.com/2020/12/16/google_europe_outage/
- IBM Cloud 2020: https://www.bleepingcomputer.com/news/technology/ibm-cloud-g...
- Cloudflare 2019: https://news.ycombinator.com/item?id=20262214
- Amazon 2018: https://www.techtarget.com/searchsecurity/news/252439945/BGP...
- AWS: https://www.thousandeyes.com/blog/route-leak-causes-amazon-a... (2015)
- Youtube: https://www.infoworld.com/article/2648947/youtube-outage-und... (2008)
And then there are incidents caused by hijacking: https://en.wikipedia.org/wiki/BGP_hijacking#:~:text=end%20us...