Alert About Missile Bound for Hawaii Was Sent in Error, Officials Say

Re:Alert About Missile Bound for Hawaii Was Sent in Error, O... by packetslave on Saturday January 13, 2018 @04:09PM (#16141421)

BINGO. If someone makes a mistake like this, unless it was deliberate or negligent (e.g. he/she didn't follow established procedures), you don't fire them! You fix the technical or procedural failures that allowed them to make the mistake in the first place. Heck, you should probably THANK them for making the system better by uncovering a bug!

When I was still mostly a Noogler, I took a large portion of Google completely down worldwide for 6 minutes. Guess what? We did a postmortem, fixed the procedural and technical problems that allowed it to happen, and I'm still here!

[ Parent | on HN ]

Re:Alert About Missile Bound for Hawaii Was Sent in Error, O... by simonebrunozzi on Saturday January 13, 2018 @07:51PM (#16142275)

This was (probably still is) the best approach at AWS in the early days (source: I worked there from 2008 to 2014). Never blame the person, find out why he/she did the mistake and make sure it doesn't happen again.

[ Parent | on HN ]

Re:Alert About Missile Bound for Hawaii Was Sent in Error, O... by sa46 on Saturday January 13, 2018 @11:58PM (#16142869)

An interesting variation to this thought is:

When is appropriate to fire someone for making a mistake?

Some ideas:

- gross negligence / malice

- ethics

- publicity damage / internet pitchforks

Additionally, it doesn't seem like the forgive mistakes approach meshes well with the startup advice of "it's never too early to fire."

[ Parent | on HN ]

Re:Alert About Missile Bound for Hawaii Was Sent in Error, O... by Renaud on Sunday January 14, 2018 @01:08AM (#16143036)

Incompetence should probably be number one.

There is a big difference between someone making a mistake due to unclear or incomplete process and someone doing that mistake because they are not competent to perform that task.

This then raises the issue that hiring and leaving incompetent people in place is itself proof of an organisational incompetence...

[ Parent | on HN ]

Re:Alert About Missile Bound for Hawaii Was Sent in Error, O... by nodesocket on Saturday January 13, 2018 @04:35PM (#16141551)

Love to hear the story how you took down Google for six minutes? Was this a long time ago?

[ Parent | on HN ]

Re:Alert About Missile Bound for Hawaii Was Sent in Error, O... by packetslave on Saturday January 13, 2018 @04:43PM (#16141583)

This was in 2012. Let's just say that a router config only getting partially applied (because of a software bug) can cause a LOT of traffic to suddenly go places it's not supposed to go.

[ Parent | on HN ]

Re:Alert About Missile Bound for Hawaii Was Sent in Error, O... by ronald_raygun on Saturday January 13, 2018 @04:48PM (#16141613)

Oh wow, I was a noogler there 4ish years ago, and they still tell that story as an example of "its okay to break things, we care about fixing them after so they dont break in the future"

[ Parent | on HN ]

Re:Alert About Missile Bound for Hawaii Was Sent in Error, O... by packetslave on Saturday January 13, 2018 @07:43PM (#16142257)

There's a couple of good stories about massive outages and good incident response. Mine is just one of them (and at some level, I was very lucky).

There's also the one where all the frontend servers worldwide went into a crash loop from a bad configuration push. The SRE doing the push noticed some "weirdness" and rolled back even before the full scope of the issue was known. That one's in the SRE book.

[ Parent | on HN ]

Re:Alert About Missile Bound for Hawaii Was Sent in Error, O... by Bluestrike2 on Saturday January 13, 2018 @08:08PM (#16142307)

Site Reliability Engineering.[0] Google's SRE book is a pretty interesting read.

0. https://landing.google.com/sre/interview/ben-treynor.html

[ Parent | on HN ]

Re:Alert About Missile Bound for Hawaii Was Sent in Error, O... by mynewtb on Saturday January 13, 2018 @07:53PM (#16142279)

GFE? SRE?

[ Parent | on HN ]

Re:Alert About Missile Bound for Hawaii Was Sent in Error, O... by mikejb on Sunday January 14, 2018 @04:32AM (#16143481)

SRE is Site Reliability Engineer; GFE is the "Google Front End"

[ Parent | on HN ]

Re:Alert About Missile Bound for Hawaii Was Sent in Error, O... by grivescorbett on Saturday January 13, 2018 @05:04PM (#16141677)

Was a noogler about a year ago. Story still told.

[ Parent | on HN ]

Re:Alert About Missile Bound for Hawaii Was Sent in Error, O... by aalleavitch on Saturday January 13, 2018 @06:30PM (#16141975)

It appears we have a living legend here!

[ Parent | on HN ]

Re:Alert About Missile Bound for Hawaii Was Sent in Error, O... by ianai on Saturday January 13, 2018 @09:05PM (#16142445)

They made him a slave of the packets though.

[ Parent | on HN ]

Re:Alert About Missile Bound for Hawaii Was Sent in Error, O... by packetslave on Saturday January 13, 2018 @07:45PM (#16142263)

I think my favorite quote about my outage was "boy, it's a good thing you didn't decide to go take a crap right about then..."

[ Parent | on HN ]

Re:Alert About Missile Bound for Hawaii Was Sent in Error, O... by ceph_ on Saturday January 13, 2018 @11:24PM (#16142800)

db31.lpp01 lol

[ Parent | on HN ]

Re:Alert About Missile Bound for Hawaii Was Sent in Error, O... by packetslave on Sunday January 14, 2018 @12:41AM (#16142978)

When they decommed it, I asked if I could have db31 to go Office Space on. They said no :-(

[ Parent | on HN ]

Re:Alert About Missile Bound for Hawaii Was Sent in Error, O... by digi_owl on Saturday January 13, 2018 @04:24PM (#16141492)

Makes me think of the stories of the early days of school computing, and how the admin would give the kids that hacked said computers part time jobs as they had demonstrated themselves both interested and talented.

[ Parent | on HN ]

Re:Alert About Missile Bound for Hawaii Was Sent in Error, O... by nfread on Sunday January 14, 2018 @11:38AM (#16144912)

They say worker hit real button rather than test button. Some folks don’t believe this b/c test button is hit on 1st of month. It is not the 1st of month so why was any button hit, especially one that is so major, how could be accident?

[ Parent | on HN ]

Re:Alert About Missile Bound for Hawaii Was Sent in Error, O... by scoggs on Saturday January 13, 2018 @06:31PM (#16141980)

Oof! I'm not envious of the way you must have felt when you realized what had happened. We've all (probably) had the same sort of feeling on a much, much smaller scale and I've never been one to enjoy that heart-sinking-into-my-shoes feeling!

[ Parent | on HN ]