News for hackers. Stuff that matters.
Re:Summary of the Amazon S3 Service Disruption by oblio on Thursday March 02, 2017 @12:28PM (#13776048)
No! Off with this head!!!

On a more serious note, if you've never done something like this, you haven't had enough interesting projects.

I've had a decent career and I still managed to:

* re-deploy the current application version in all our data centers, instead of the new version, in a period when our deployment wasn't a 0-downtime one

* rename all the Jenkins jobs on the server to the same name, thus deleting hundreds of Jenkins jobs in one fell swoop

"Let him who is without sin cast the first stone" and all that :)

Re:Summary of the Amazon S3 Service Disruption by ckozlowski on Thursday March 02, 2017 @12:38PM (#13776178)
There's two things I've come to believe in my IT career regarding operations:

1. No organization anywhere is a paragon of excellence, and everyone can benefit from improvement. 2. Every organization is made up of humans just like you. With all that entails.

Some things which seem blatantly obvious after the fact are easily overlooked when the pressure to deliver is high and other issues are taking precedence.

I completely agree with your statement. In fact, when I do interviews, one of my favorite and most insightful questions to ask is, basically, "tell me about a time you screwed the pooch." If they don't have a story and they worked in ops, then it can suggest they didn't really do much. The really sharp ones I've interviewed have a good story or two (and can tell it in excruciating detail. =)

* At a prior company I once tried appending to the list of NFS exports, but dropped the "no-root-squash" option, and instantly denied write permissions to our entire VMware farm. You can imagine what then happened to all of the VMs for this mission critical customer. =P

Re:Summary of the Amazon S3 Service Disruption by tetha on Thursday March 02, 2017 @01:08PM (#13776530)
As I keep saying, there are people who screw up big time and people who are too scared to touch the system. I managed to gun down three productive clusters by deploying NTP by accident along with a tiny change. Kerpow, 12 minutes of downtime, full network outage due to DHCP and such. Great fun.