Meh, that's a process problem, not a people problem. Playbooks that have you retype commands with complex options, with no confirmation, etc, are inviting that sort of thing.

They just spent millions of dollars training that person to never make this mistake again, should definitely keep them around.

The wording of the article implies Amazon is shifting the blame entirely on the individual who typo'd: they indemnify themselves with "an authorized S3 team member using an established playbook..." ("don't blame us, our process is perfect!")

There are process-fixes for this, such as requiring a two-person rule when at a production shell and modifying tooling to detect potentially unintentional commands (e.g. a SQL UPDATE without a WHERE) - but given what I know about Amazon's internal practices (i.e. the brutality) it wouldn't surprise me if they did terminate the unfortunate operator - not because they want to, but because AWS simply has too many large-scale customers who would demand immediate action like that.

Not at all. The "an authorized S3 team member using an established playbook..." is to establish that:

1. They were authorized to be doing it. 2. They were following an established process, and not winging it.

The reason for that wording is to illustrate that operational changes are made in accordance with established CM.

Mistakes happen. The system didn't catch the error. That's why the mitigation is "fix the system", and not "fire the team member". =)

Maybe so. I didn't take it that way. I took it as, "nobody panic, we weren't hacked."

It's both. This isn't "our system was compromised by attack;" it's "SNAFU."

Everyone who's had operations experience knows that there will be, as time approaches infinity, more than zero SNAFU. That's why companies offer five nines of uptime, not 100% uptime.

I brought down our production system after a typo in a command once... the dev team took the blame for allowing an illegal parameter to bring down the system.

Me too. I accidentally SSH'd into the wrong data center and ran a for loop restarting every database server once.

Fixed that by putting the DC domain in red on the prompt.

I did once, too. Unfortunately, it was company of 4 people so I was also the dev team. :-) Never did it again though (yet).

Kudos to the dev team for that, I think most people wouldn't own that kind of issue

It was a very well-run engineering department where taking blame was not a career ending decision. I took full blame for the typo (at 3am trying to resolve a customer issue), but the dev team accepted full responsibility for letting it take down the system.

Every mistake was used as a learning opportunity to ensure that the same and similar mistakes can't be repeated.

That person is the least likely to make that mistake again!

Reminds me of the story about Bob Hoover.[1]

He took off in his piston engine plane, only to lose power during the climb and was forced to make a crash landing. It turned out the airplane was fueled with jet fuel instead of regular gasoline (the ground crewman mistakenly thought the plane was a turbo prop).

Instead of yelling at or firing the ground crewman, Hoover had this to say[2]:

    "There isn't a man alive who hasn't made a mistake.
    But I'm positive you'll never make this mistake again.
    That's why I want to make sure that you're the only one
    to refuel my plane tomorrow. I won't let anyone else
    on the field touch it."

[1] https://www.aopa.org/news-and-media/all-news/2014/july/pilot...

[2] http://www.squawkpoint.com/2014/01/criticism/

No! Off with this head!!!

On a more serious note, if you've never done something like this, you haven't had enough interesting projects.

I've had a decent career and I still managed to:

* re-deploy the current application version in all our data centers, instead of the new version, in a period when our deployment wasn't a 0-downtime one

* rename all the Jenkins jobs on the server to the same name, thus deleting hundreds of Jenkins jobs in one fell swoop

"Let him who is without sin cast the first stone" and all that :)

There's two things I've come to believe in my IT career regarding operations:

1. No organization anywhere is a paragon of excellence, and everyone can benefit from improvement. 2. Every organization is made up of humans just like you. With all that entails.

Some things which seem blatantly obvious after the fact are easily overlooked when the pressure to deliver is high and other issues are taking precedence.

I completely agree with your statement. In fact, when I do interviews, one of my favorite and most insightful questions to ask is, basically, "tell me about a time you screwed the pooch." If they don't have a story and they worked in ops, then it can suggest they didn't really do much. The really sharp ones I've interviewed have a good story or two (and can tell it in excruciating detail. =)

* At a prior company I once tried appending to the list of NFS exports, but dropped the "no-root-squash" option, and instantly denied write permissions to our entire VMware farm. You can imagine what then happened to all of the VMs for this mission critical customer. =P

As I keep saying, there are people who screw up big time and people who are too scared to touch the system. I managed to gun down three productive clusters by deploying NTP by accident along with a tiny change. Kerpow, 12 minutes of downtime, full network outage due to DHCP and such. Great fun.

At these scales it's the fault of the system, not the individual, so hopefully they don't come down hard on them.

Agreed. They also seemed to acknowledge that in the post, as they mentioned improving the tool to not allow such destructive options.

You mean like range checking the input parameters to the command? =)

    SHUT DOWN S3?  ARE YOU SURE? (y/N) :

Were it so easy. Likely more like

"Shut down 73 servers? Are you sure? (y/N)":

"Seventy-three? Wow, I hadn't realized our system grew that much. Probably a new backend dependency got added that I'm not familiar with yet; I'll look into it later." (Y)

Every sysadmin at my previous job (a Fortune 500) would stop the moment that number is off the expected by a few machines, just long enough to verify it's correct. That may be due to having made a mistake like this once. I know that's true in several other large shops, as well.

Source: I'm was the one the one they would call for our team... usually at 4:00 AM because one of our team members (which was also frequently me) didn't document something correctly.

More like:

SHUT DOWN S3? ARE YOU SURE? ("I'm absolutely positive."/n) :

Too easy. should make u run NP-hard solver to confirm.

I would love to hear their recollection of the day. Would make me feel better about my mistakes.