News for hackers. Stuff that matters.
Re:Google Cloud Networking Incident Postmortem by karlding on Friday June 07, 2019 @01:30AM (#20121891)
I don't work as an SRE, but isn't that covered by providing engineers physical access to secure facilities in the absolute worst case?

The article states:

> The defense in depth philosophy means we have robust backup plans for handling failure of such tools, but use of these backup plans (including engineers travelling to secure facilities designed to withstand the most catastrophic failures, and a reduction in priority of less critical network traffic classes to reduce congestion) added to the time spent debugging.

Re:Google Cloud Networking Incident Postmortem by log_n on Friday June 07, 2019 @10:58AM (#20125202)
You don't have to go that far. You could also have automated roll-back of individual servers if they sense something is off, for instance.

Another alternative is low bandwidth flag based roll-backs (for instances such as this where the network is congested but not completely lost).