Re:Summary of the Amazon S3 Service Disruption
by ckozlowski on Thursday March 02, 2017 @01:25PM
(#13776702)
Not at all. The "an authorized S3 team member using an established playbook..." is to establish that:
1. They were authorized to be doing it. 2. They were following an established process, and not winging it.
The reason for that wording is to illustrate that operational changes are made in accordance with established CM.
Mistakes happen. The system didn't catch the error. That's why the mitigation is "fix the system", and not "fire the team member". =)
Re:Summary of the Amazon S3 Service Disruption
by unit91 on Thursday March 02, 2017 @01:15PM
(#13776607)
Maybe so. I didn't take it that way. I took it as, "nobody panic, we weren't hacked."
Re:Summary of the Amazon S3 Service Disruption
by fixermark on Thursday March 02, 2017 @01:29PM
(#13776748)
It's both. This isn't "our system was compromised by attack;" it's "SNAFU."
Everyone who's had operations experience knows that there will be, as time approaches infinity, more than zero SNAFU. That's why companies offer five nines of uptime, not 100% uptime.
There are process-fixes for this, such as requiring a two-person rule when at a production shell and modifying tooling to detect potentially unintentional commands (e.g. a SQL UPDATE without a WHERE) - but given what I know about Amazon's internal practices (i.e. the brutality) it wouldn't surprise me if they did terminate the unfortunate operator - not because they want to, but because AWS simply has too many large-scale customers who would demand immediate action like that.