News for hackers. Stuff that matters.
Re:Google Cloud Networking Incident Postmortem by ethbro on Friday June 07, 2019 @01:08AM (#20121807)
I've only been tangentially pulled into high severity incidents, but the thing that most impressed me was the quiet.

As mentioned in this thread, it's a lot like listening to air traffic comm chatter.

People say what they know, and only what they know, and clearly identify anything they're unsure about. Informative and clear communication matters more than brilliance.

Most of the traffic is async task identification, dispatch, and then reporting in.

And if anyone is screaming or gets emotional, they should not be in that room.

Re:Google Cloud Networking Incident Postmortem by tetha on Friday June 07, 2019 @01:49AM (#20121962)
Someone at our place recently commented that the ops team during an incident strongly feels like NASA mission control in critical moments[1]. I wanted to protest, but that's surprisingly accurate.

> And if anyone is screaming or gets emotional, they should not be in that room.

If someone starts yelling around in my incident war room for no reason, they get thrown out. I'm a calm and quiet person, but bugging around during a major incident is one of the few things that make me mad.

1: https://youtu.be/Y0yOTanzx-s?t=3059

Re:Google Cloud Networking Incident Postmortem by di4na on Friday June 07, 2019 @08:07AM (#20123692)
It is not surprising at all. Mission Control was forged in the fire (literally for Apollo 1) and they are one of the most visible "incident team" we know about.

I highly advise to read Gene Kranz memoirs "Failure is not an Option" if you work in that kind of environment.

Re:Google Cloud Networking Incident Postmortem by ethbro on Friday June 07, 2019 @11:10AM (#20125318)
I heard recently that he never said that.

Apparently it was mentioned when the Apollo 13 script writers were gathering stories at NASA, they liked it, and then gave it to the Kranz character.

Who then decided, "Hey, if everyone thinks I said it..." and titled his memoir.

Re:Google Cloud Networking Incident Postmortem by di4na on Friday June 07, 2019 @02:30PM (#20127376)
Yep exactly.
Re:Google Cloud Networking Incident Postmortem by Moru on Friday June 07, 2019 @04:29AM (#20122678)
When the insident is over, does it look like 55:50 in that video? :-)
Re:Google Cloud Networking Incident Postmortem by tetha on Friday June 07, 2019 @07:13PM (#20129459)
We recently had a 15 month long project almost fail due to some stupid shit and a really wonky error no one understood so far. <Almost fail> as in "Keep the customer on the phone we have 3 possible things to try and don't hang up! No one leaves that call until I'm out of hacks to deploy!" That evening we had the entire ops team in the houston-mode for several hours.

And yes, once we had a workaround in place the customer accepted, we reacted like that. Except we also had our critical-incident whiskey go around. Then the CEO walked in to congratulate us on that project. Whoops. But he's a good sport, so good times. :)