At some point people realized multi-server systems within one AZ are prone to failure. They then started deploying their system redundantly to multiple AZs within the same region to increase availability. This helped, but created consistency issues. To fix this people started building multi-AZ software systems, creating dependencies across AZs that weakened overall availability.
At some point people realized multi-AZ systems within one region are prone to failure. They then started deploying their system redundantly to multiple regions of the same cloud platform to increase availability. This helped, but created consistency issues. To fix this people started building multi-region software systems, creating dependencies across regions that weakened overall availability.
At some point people realized multi-region systems within one cloud platform are prone to failure. They then started deploying their system redundantly to multiple cloud platforms to increase availability. This helped, but created consistency issues. To fix this people started building multi-cloud software systems, creating dependencies across cloud platforms that weakened overall availability.
But no, cloud everything.
Man that's got to suck.
I HAVE NO TOOLS BECAUSE I’VE DESTROYED MY TOOLS WITH MY TOOLS.A very simple example, you do something stupid on a remote machine (either high network usage or CPU usage) over SSH then you can't undo it because SSH becomes unresponsive
Bohr bugs generate an alert and happily meander through normal support channels.
Heisenbugs go through phases -
1. Probation. On continued failure,
2. Restart. If the app or service fails after a restart,
3. Reboot. If the app or service fails after a reboot,
4. Re-image. If the app or service fails after re-imaging,
5. Remove/elimate the node.
The article states:
> The defense in depth philosophy means we have robust backup plans for handling failure of such tools, but use of these backup plans (including engineers travelling to secure facilities designed to withstand the most catastrophic failures, and a reduction in priority of less critical network traffic classes to reduce congestion) added to the time spent debugging.
Another alternative is low bandwidth flag based roll-backs (for instances such as this where the network is congested but not completely lost).
My point being, it seems that modems are becoming less-and-less viable for out-of-band management.
[0] https://scholar.harvard.edu/files/mickens/files/thesaddestmo...
Answer, and the root cause summarized:
Maintenance started in a physical location, and then "... the automation software created a list of jobs to deschedule in that physical location, which included the logical clusters running network control jobs. Those logical clusters also included network control jobs in other physical locations."
So the automation equivalent of a human driven command that says "deschedule these core jobs in another region".
Maybe someone needs to write a paper on Fault tolerance in the presence of Byzantine Automations (Joke. There was a satirical note on this subject posted here yesterday.)