Re:Google Cloud Networking Incident Postmortem
by londons_explore on Friday June 07, 2019 @12:55AM
(#20121754)
It isn't a worst-case though. They should have had the capability to resolve this issue with no network connectivity, which would be the worst case failure of the network control plane.
[ Parent
| on HN ]
Re:Google Cloud Networking Incident Postmortem
by karlding on Friday June 07, 2019 @01:30AM
(#20121891)
I don't work as an SRE, but isn't that covered by providing engineers physical access to secure facilities in the absolute worst case?
The article states:
> The defense in depth philosophy means we have robust backup plans for handling failure of such tools, but use of these backup plans (including engineers travelling to secure facilities designed to withstand the most catastrophic failures, and a reduction in priority of less critical network traffic classes to reduce congestion) added to the time spent debugging.
Re:Google Cloud Networking Incident Postmortem
by log_n on Friday June 07, 2019 @10:58AM
(#20125202)
You don't have to go that far. You could also have automated roll-back of individual servers if they sense something is off, for instance.
Another alternative is low bandwidth flag based roll-backs (for instances such as this where the network is congested but not completely lost).
Re:Google Cloud Networking Incident Postmortem
by smileypete on Friday June 07, 2019 @06:00AM
(#20123089)
Use a 28k modem? :)
Re:Google Cloud Networking Incident Postmortem
by _n_b_ on Friday June 07, 2019 @07:44AM
(#20123535)
An anecdote: my (not-IT) company does exactly this for out-of-band management... except, in one small satellite location, the phone company no longer provided any copper POTS lines; all they could do was an RJ-11 jack out of the ONT that was backhauled as (lossy) VoIP. So the modem couldn't be made to work.
My point being, it seems that modems are becoming less-and-less viable for out-of-band management.
Re:Google Cloud Networking Incident Postmortem
by Bluecobra on Friday June 07, 2019 @08:50AM
(#20123999)
Fun story, AT&T forced our hands to get off our PRI (voice T1) and move to their fiber service. They also insisted that they have a dedicated phone line installed so they can dial into their modem in case of circuit failure. We can’t buy a cooper phone line from them, so it gets routed over the same fiber circuit and goes to a digital to analog device back to the router. I don’t think one hand talks to the other over there...