News for hackers. Stuff that matters.

A comment on Alert About Missile Bound for Hawaii Was Sent in Error, Officials Say. Parent comment

Re:Alert About Missile Bound for Hawaii Was Sent in Error, O... by nodesocket on Saturday January 13, 2018 @04:35PM (#16141551)
Love to hear the story how you took down Google for six minutes? Was this a long time ago?
[ Parent | on HN ]
Re:Alert About Missile Bound for Hawaii Was Sent in Error, O... by packetslave on Saturday January 13, 2018 @04:43PM (#16141583)
This was in 2012. Let's just say that a router config only getting partially applied (because of a software bug) can cause a LOT of traffic to suddenly go places it's not supposed to go.
[ Parent | on HN ]
Re:Alert About Missile Bound for Hawaii Was Sent in Error, O... by ronald_raygun on Saturday January 13, 2018 @04:48PM (#16141613)
Oh wow, I was a noogler there 4ish years ago, and they still tell that story as an example of "its okay to break things, we care about fixing them after so they dont break in the future"
[ Parent | on HN ]
Re:Alert About Missile Bound for Hawaii Was Sent in Error, O... by packetslave on Saturday January 13, 2018 @07:43PM (#16142257)
There's a couple of good stories about massive outages and good incident response. Mine is just one of them (and at some level, I was very lucky).

There's also the one where all the frontend servers worldwide went into a crash loop from a bad configuration push. The SRE doing the push noticed some "weirdness" and rolled back even before the full scope of the issue was known. That one's in the SRE book.

[ Parent | on HN ]
Re:Alert About Missile Bound for Hawaii Was Sent in Error, O... by Bluestrike2 on Saturday January 13, 2018 @08:08PM (#16142307)
Site Reliability Engineering.[0] Google's SRE book is a pretty interesting read.

0. https://landing.google.com/sre/interview/ben-treynor.html

[ Parent | on HN ]
Re:Alert About Missile Bound for Hawaii Was Sent in Error, O... by mynewtb on Saturday January 13, 2018 @07:53PM (#16142279)
GFE? SRE?
[ Parent | on HN ]
Re:Alert About Missile Bound for Hawaii Was Sent in Error, O... by mikejb on Sunday January 14, 2018 @04:32AM (#16143481)
SRE is Site Reliability Engineer; GFE is the "Google Front End"
[ Parent | on HN ]
Re:Alert About Missile Bound for Hawaii Was Sent in Error, O... by grivescorbett on Saturday January 13, 2018 @05:04PM (#16141677)
Was a noogler about a year ago. Story still told.
[ Parent | on HN ]
Re:Alert About Missile Bound for Hawaii Was Sent in Error, O... by aalleavitch on Saturday January 13, 2018 @06:30PM (#16141975)
It appears we have a living legend here!
[ Parent | on HN ]
Re:Alert About Missile Bound for Hawaii Was Sent in Error, O... by ianai on Saturday January 13, 2018 @09:05PM (#16142445)
They made him a slave of the packets though.
[ Parent | on HN ]
Re:Alert About Missile Bound for Hawaii Was Sent in Error, O... by packetslave on Saturday January 13, 2018 @07:45PM (#16142263)
I think my favorite quote about my outage was "boy, it's a good thing you didn't decide to go take a crap right about then..."
[ Parent | on HN ]
Re:Alert About Missile Bound for Hawaii Was Sent in Error, O... by ceph_ on Saturday January 13, 2018 @11:24PM (#16142800)
db31.lpp01 lol
[ Parent | on HN ]
Re:Alert About Missile Bound for Hawaii Was Sent in Error, O... by packetslave on Sunday January 14, 2018 @12:41AM (#16142978)
When they decommed it, I asked if I could have db31 to go Office Space on. They said no :-(
[ Parent | on HN ]