News for hackers. Stuff that matters.
Re:Google Cloud Networking Incident Postmortem by tetha on Friday June 07, 2019 @01:49AM (#20121962)
Someone at our place recently commented that the ops team during an incident strongly feels like NASA mission control in critical moments[1]. I wanted to protest, but that's surprisingly accurate.

> And if anyone is screaming or gets emotional, they should not be in that room.

If someone starts yelling around in my incident war room for no reason, they get thrown out. I'm a calm and quiet person, but bugging around during a major incident is one of the few things that make me mad.

1: https://youtu.be/Y0yOTanzx-s?t=3059

Re:Google Cloud Networking Incident Postmortem by di4na on Friday June 07, 2019 @08:07AM (#20123692)
It is not surprising at all. Mission Control was forged in the fire (literally for Apollo 1) and they are one of the most visible "incident team" we know about.

I highly advise to read Gene Kranz memoirs "Failure is not an Option" if you work in that kind of environment.

Re:Google Cloud Networking Incident Postmortem by ethbro on Friday June 07, 2019 @11:10AM (#20125318)
I heard recently that he never said that.

Apparently it was mentioned when the Apollo 13 script writers were gathering stories at NASA, they liked it, and then gave it to the Kranz character.

Who then decided, "Hey, if everyone thinks I said it..." and titled his memoir.

Re:Google Cloud Networking Incident Postmortem by di4na on Friday June 07, 2019 @02:30PM (#20127376)
Yep exactly.
Re:Google Cloud Networking Incident Postmortem by Moru on Friday June 07, 2019 @04:29AM (#20122678)
When the insident is over, does it look like 55:50 in that video? :-)
Re:Google Cloud Networking Incident Postmortem by tetha on Friday June 07, 2019 @07:13PM (#20129459)
We recently had a 15 month long project almost fail due to some stupid shit and a really wonky error no one understood so far. <Almost fail> as in "Keep the customer on the phone we have 3 possible things to try and don't hang up! No one leaves that call until I'm out of hacks to deploy!" That evening we had the entire ops team in the houston-mode for several hours.

And yes, once we had a workaround in place the customer accepted, we reacted like that. Except we also had our critical-incident whiskey go around. Then the CEO walked in to congratulate us on that project. Whoops. But he's a good sport, so good times. :)