Re:Google Cloud Networking Incident Postmortem
by di4na on Friday June 07, 2019 @08:07AM
(#20123692)
It is not surprising at all. Mission Control was forged in the fire (literally for Apollo 1) and they are one of the most visible "incident team" we know about.
I highly advise to read Gene Kranz memoirs "Failure is not an Option" if you work in that kind of environment.
Re:Google Cloud Networking Incident Postmortem
by ethbro on Friday June 07, 2019 @11:10AM
(#20125318)
I heard recently that he never said that.
Apparently it was mentioned when the Apollo 13 script writers were gathering stories at NASA, they liked it, and then gave it to the Kranz character.
Who then decided, "Hey, if everyone thinks I said it..." and titled his memoir.
Re:Google Cloud Networking Incident Postmortem
by di4na on Friday June 07, 2019 @02:30PM
(#20127376)
Yep exactly.
When the insident is over, does it look like 55:50 in that video? :-)
Re:Google Cloud Networking Incident Postmortem
by tetha on Friday June 07, 2019 @07:13PM
(#20129459)
We recently had a 15 month long project almost fail due to some stupid shit and a really wonky error no one understood so far. <Almost fail> as in "Keep the customer on the phone we have 3 possible things to try and don't hang up! No one leaves that call until I'm out of hacks to deploy!" That evening we had the entire ops team in the houston-mode for several hours.
And yes, once we had a workaround in place the customer accepted, we reacted like that. Except we also had our critical-incident whiskey go around. Then the CEO walked in to congratulate us on that project. Whoops. But he's a good sport, so good times. :)
> And if anyone is screaming or gets emotional, they should not be in that room.
If someone starts yelling around in my incident war room for no reason, they get thrown out. I'm a calm and quiet person, but bugging around during a major incident is one of the few things that make me mad.
1: https://youtu.be/Y0yOTanzx-s?t=3059