News for hackers. Stuff that matters.

A comment on Summary of the Amazon S3 Service Disruption. Parent comment

Re:Summary of the Amazon S3 Service Disruption by Johnny555 on Thursday March 02, 2017 @12:26PM (#13776026)
I brought down our production system after a typo in a command once... the dev team took the blame for allowing an illegal parameter to bring down the system.
[ Parent | on HN ]
Re:Summary of the Amazon S3 Service Disruption by vhost- on Thursday March 02, 2017 @04:27PM (#13778398)
Me too. I accidentally SSH'd into the wrong data center and ran a for loop restarting every database server once.

Fixed that by putting the DC domain in red on the prompt.

[ Parent | on HN ]
Re:Summary of the Amazon S3 Service Disruption by unit91 on Thursday March 02, 2017 @01:17PM (#13776622)
I did once, too. Unfortunately, it was company of 4 people so I was also the dev team. :-) Never did it again though (yet).
[ Parent | on HN ]
Re:Summary of the Amazon S3 Service Disruption by alexvy86 on Thursday March 02, 2017 @12:38PM (#13776183)
Kudos to the dev team for that, I think most people wouldn't own that kind of issue
[ Parent | on HN ]
Re:Summary of the Amazon S3 Service Disruption by Johnny555 on Thursday March 02, 2017 @12:42PM (#13776230)
It was a very well-run engineering department where taking blame was not a career ending decision. I took full blame for the typo (at 3am trying to resolve a customer issue), but the dev team accepted full responsibility for letting it take down the system.

Every mistake was used as a learning opportunity to ensure that the same and similar mistakes can't be repeated.

[ Parent | on HN ]