News for hackers. Stuff that matters.
Re:Google Cloud Networking Incident Postmortem by carlsborg on Thursday June 06, 2019 @09:41PM (#20120924)
I was curious to know how cascading failures in one region effected other regions. Impact was " ...increased latency, intermittent errors, and connectivity loss to instances in us-central1, us-east1, us-east4, us-west2, northamerica-northeast1, and southamerica-east1."

Answer, and the root cause summarized:

Maintenance started in a physical location, and then "... the automation software created a list of jobs to deschedule in that physical location, which included the logical clusters running network control jobs. Those logical clusters also included network control jobs in other physical locations."

So the automation equivalent of a human driven command that says "deschedule these core jobs in another region".

Maybe someone needs to write a paper on Fault tolerance in the presence of Byzantine Automations (Joke. There was a satirical note on this subject posted here yesterday.)

Re:Google Cloud Networking Incident Postmortem by mslot on Friday June 07, 2019 @08:23AM (#20123823)
At some point people realized servers are prone to failure. They then started deploying their system redundantly to multiple servers in the data center (AZ) to increase availability. This helped, but created consistency issues. To fix this people started building multi-server software systems, creating dependencies across servers that weakened overall availability.

At some point people realized multi-server systems within one AZ are prone to failure. They then started deploying their system redundantly to multiple AZs within the same region to increase availability. This helped, but created consistency issues. To fix this people started building multi-AZ software systems, creating dependencies across AZs that weakened overall availability.

At some point people realized multi-AZ systems within one region are prone to failure. They then started deploying their system redundantly to multiple regions of the same cloud platform to increase availability. This helped, but created consistency issues. To fix this people started building multi-region software systems, creating dependencies across regions that weakened overall availability.

At some point people realized multi-region systems within one cloud platform are prone to failure. They then started deploying their system redundantly to multiple cloud platforms to increase availability. This helped, but created consistency issues. To fix this people started building multi-cloud software systems, creating dependencies across cloud platforms that weakened overall availability.

Re:Google Cloud Networking Incident Postmortem by log_n on Friday June 07, 2019 @11:57AM (#20125856)
Humanity's colonization of the stars will eventually be spurred on by the desire improve the resiliency of SaaS.
Re:Google Cloud Networking Incident Postmortem by asark on Friday June 07, 2019 @12:39PM (#20126380)
I honestly think a one- or two-server setup with scripted server re-creation, proven in Vagrant or whatever, and maybe using Docker but only to isolate services on those one or two servers and make it easier to re-create, tested frequently by spinning up local dev copies, and (obviously) with backups, is probably a stabler, cheaper, and higher-availability set-up for the vast majority of use cases. Even if predicted, scheduled downtime is somewhat higher, it's probably worth it for the many benefits.

But no, cloud everything.

Re:Google Cloud Networking Incident Postmortem by ohum on Saturday June 08, 2019 @01:24PM (#20133677)
At some point the system realized people are prone to failure. It then started deploying itself redundantly without dependence on people. This helped, but the people interfered. To fix this the system started building self replicating physical instances, creating new dependencies on material that resulted in grey goo (or paper clips).
Re:Google Cloud Networking Incident Postmortem by dnautics on Friday June 07, 2019 @12:08AM (#20121594)
> Debugging the problem was significantly hampered by failure of tools competing over use of the now-congested network.

Man that's got to suck.

Re:Google Cloud Networking Incident Postmortem by abbeyj on Friday June 07, 2019 @11:27AM (#20125511)
This reminded me of https://www.usenix.org/system/files/1311_05-08_mickens.pdf

   I HAVE NO TOOLS BECAUSE I’VE DESTROYED MY TOOLS WITH MY TOOLS.
Re:Google Cloud Networking Incident Postmortem by tbodt on Friday June 07, 2019 @09:16PM (#20129996)
You joke, but I've read a Google postmortem with that exact quote at the top. This incident would be more fitting though.
Re:Google Cloud Networking Incident Postmortem by ikiris on Friday June 07, 2019 @10:00PM (#20130147)
It's in this one now.
Re:Google Cloud Networking Incident Postmortem by theflyinghorse on Friday June 07, 2019 @11:56AM (#20125844)
What an amazing read!
Re:Google Cloud Networking Incident Postmortem by raverbashing on Friday June 07, 2019 @02:22AM (#20122104)
This happens way more often than you think

A very simple example, you do something stupid on a remote machine (either high network usage or CPU usage) over SSH then you can't undo it because SSH becomes unresponsive

Re:Google Cloud Networking Incident Postmortem by isatty on Friday June 07, 2019 @04:40AM (#20122737)
Yeah got locked out of a dedi this way (bad ipfw ruleset which killed my ssh connection) and the virtual console wasn’t working. Fun times. At least it was a personal machine.
Re:Google Cloud Networking Incident Postmortem by Cthulhu_ on Friday June 07, 2019 @03:37AM (#20122423)
Is there anything that can be done to mitigate that? E.g. give ssh network and the daemon top cpu priority?
Re:Google Cloud Networking Incident Postmortem by Spearchucker on Friday June 07, 2019 @04:00AM (#20122542)
Recovery-oriented computing. Seperate Bohr bugs (easy to reproduce) from Heisenbugs (difficult to reproduce/change their behaviour once investigated).

Bohr bugs generate an alert and happily meander through normal support channels.

Heisenbugs go through phases -

1. Probation. On continued failure,

2. Restart. If the app or service fails after a restart,

3. Reboot. If the app or service fails after a reboot,

4. Re-image. If the app or service fails after re-imaging,

5. Remove/elimate the node.

Re:Google Cloud Networking Incident Postmortem by vorpalhex on Friday June 07, 2019 @10:00AM (#20124543)
Some equipment will auto-revert to a last known good configuration if you don't approve new changes within a window... though high CPU could lock that process up..
Re:Google Cloud Networking Incident Postmortem by riskrisk on Saturday June 08, 2019 @02:14AM (#20130932)
in this case the old configuration was lost, it took an hour to rebuild because tooling normally used to rebuild it for testing was unavailable and building had to be done locally, using a single machine someone ssh-ed into, and that just takes a while. Luckily, a person was around who knew how to do the rebuild without fancy tooling.
Re:Google Cloud Networking Incident Postmortem by bpchaps on Friday June 07, 2019 @08:28AM (#20123857)
Yes - use taskset or isolcpus with other magic to put sshd on its own CPU core, or one core per CPU. Lots of HFT places do that.
Re:Google Cloud Networking Incident Postmortem by dnautics on Sunday June 09, 2019 @03:13AM (#20137699)
That doesn't help if the problem is a bandwidth congestion problem.
Re:Google Cloud Networking Incident Postmortem by bpchaps on Sunday June 09, 2019 @10:06PM (#20143296)
It can help some amount, though. Bind the NIC interrupts to a small handful of cores. Or, ensure that ssh only works through a management NIC, and have that NIC bound to the same cores as sshd. You can get really fancy with these setups, especially when working with NUMA stuffs
Re:Google Cloud Networking Incident Postmortem by Causality1 on Friday June 07, 2019 @03:56AM (#20122517)
I'm a bit surprised there's no sort of SSH undo subroutine that reverses the previous command if connectivity is lost. Of course it couldn't cover every possible stupid thing but it could fix simple stupid mistakes like fouling up a port assignment or disabling the wrong network adapter.
Re:Google Cloud Networking Incident Postmortem by topranks on Friday June 07, 2019 @04:02AM (#20122559)
A commit / confirm / rollback cycle is how this is typically dealt with in network automation.
Re:Google Cloud Networking Incident Postmortem by wbl on Friday June 07, 2019 @09:37AM (#20124318)
How does ssh know what command you did and how to reverse it?
Re:Google Cloud Networking Incident Postmortem by londons_explore on Friday June 07, 2019 @12:55AM (#20121754)
It isn't a worst-case though. They should have had the capability to resolve this issue with no network connectivity, which would be the worst case failure of the network control plane.
Re:Google Cloud Networking Incident Postmortem by karlding on Friday June 07, 2019 @01:30AM (#20121891)
I don't work as an SRE, but isn't that covered by providing engineers physical access to secure facilities in the absolute worst case?

The article states:

> The defense in depth philosophy means we have robust backup plans for handling failure of such tools, but use of these backup plans (including engineers travelling to secure facilities designed to withstand the most catastrophic failures, and a reduction in priority of less critical network traffic classes to reduce congestion) added to the time spent debugging.

Re:Google Cloud Networking Incident Postmortem by log_n on Friday June 07, 2019 @10:58AM (#20125202)
You don't have to go that far. You could also have automated roll-back of individual servers if they sense something is off, for instance.

Another alternative is low bandwidth flag based roll-backs (for instances such as this where the network is congested but not completely lost).

Re:Google Cloud Networking Incident Postmortem by smileypete on Friday June 07, 2019 @06:00AM (#20123089)
Use a 28k modem? :)
Re:Google Cloud Networking Incident Postmortem by _n_b_ on Friday June 07, 2019 @07:44AM (#20123535)
An anecdote: my (not-IT) company does exactly this for out-of-band management... except, in one small satellite location, the phone company no longer provided any copper POTS lines; all they could do was an RJ-11 jack out of the ONT that was backhauled as (lossy) VoIP. So the modem couldn't be made to work.

My point being, it seems that modems are becoming less-and-less viable for out-of-band management.

Re:Google Cloud Networking Incident Postmortem by Bluecobra on Friday June 07, 2019 @08:50AM (#20123999)
Fun story, AT&T forced our hands to get off our PRI (voice T1) and move to their fiber service. They also insisted that they have a dedicated phone line installed so they can dial into their modem in case of circuit failure. We can’t buy a cooper phone line from them, so it gets routed over the same fiber circuit and goes to a digital to analog device back to the router. I don’t think one hand talks to the other over there...
Re:Google Cloud Networking Incident Postmortem by bnjms on Friday June 07, 2019 @09:02PM (#20129921)
If this is a problem consider Opengear.

https://opengear.com/products/acm7000-resilience-gateway

Re:Google Cloud Networking Incident Postmortem by oldcreek12 on Friday June 07, 2019 @02:38AM (#20122172)
This is inconceivable ... they don't have an OOB management network?
Re:Google Cloud Networking Incident Postmortem by kbirkeland on Friday June 07, 2019 @10:02AM (#20124571)
A completely OOB management network is an amazingly high cost when you have presence all over the world. I don't think anybody has gone to the length to double up on dark fiber and OTN gear just for management traffic.
Re:Google Cloud Networking Incident Postmortem by ddalex on Friday June 07, 2019 @11:37AM (#20125624)
Hmm... With 5G each blade in the rack could get its own modem and sim card for OOB management.
Re:Google Cloud Networking Incident Postmortem by bnjms on Friday June 07, 2019 @09:06PM (#20129939)
Why would you do that when you could have 1 sim in a 96 port terminal server?
Re:Google Cloud Networking Incident Postmortem by riskrisk on Saturday June 08, 2019 @02:22AM (#20130955)
That's less of an issue. Issue is in how you classify traffic on a network. e.g. Gmail, it's helpful to incident response, should it be used for OOB management.
Re:Google Cloud Networking Incident Postmortem by ikiris on Friday June 07, 2019 @10:02PM (#20130155)
Still have to get to it.
Re:Google Cloud Networking Incident Postmortem by KirinDave on Friday June 07, 2019 @02:30AM (#20122140)
You have no idea.
Re:Google Cloud Networking Incident Postmortem by tpaschalis on Friday June 07, 2019 @05:05AM (#20122856)
Said satirical note on Byzantine fault tolerance is on this link [0]. As usual for Mickens, gives that "funny, but true" sense.

[0] https://scholar.harvard.edu/files/mickens/files/thesaddestmo...

Re:Google Cloud Networking Incident Postmortem by carlsborg on Friday June 07, 2019 @08:38AM (#20123918)
My view on this: System engineering is as important as the algorithms. And the system engineering team should have at least some grey hair.
Re:Google Cloud Networking Incident Postmortem by hguant on Friday June 07, 2019 @09:53AM (#20124466)
Working in a systems engineering position for a year and a half now: the grey hair comes to you.
Re:Google Cloud Networking Incident Postmortem by mxuribe on Friday June 07, 2019 @10:35AM (#20124920)
Hadn't read this before; very funny...and as noted, true!