News for hackers. Stuff that matters.
Re:Summary of the Amazon S3 Service Disruption by pmoriarty on Thursday March 02, 2017 @01:51PM (#13776973)
Putting the capability to take down S3 in to the hands of a single engineer seems a bit much.

Is mere extra training the right solution here?

Maybe they need something like the procedure that's used in missile silos:

Not allowing the shutdown system to function at all without the explicit authorization of least two people.

Re:Summary of the Amazon S3 Service Disruption by notatoad on Thursday March 02, 2017 @01:56PM (#13777035)
The linked article also says the tools they use were changed to limit the amount of resources that could be taken down at a single time, the speed they could be taken down at, and a hard floor was put on the number of instances that could be stopped.

That's a lot more than just extra training, and a lot better than a two-key system.

Re:Summary of the Amazon S3 Service Disruption by ceejayoz on Thursday March 02, 2017 @02:12PM (#13777206)
> Maybe they need something like the procedure that's used in missile silos...

Probably a bad example. The system was a pain in the ass, so they went and circumvented some of its restrictions.

http://gizmodo.com/for-20-years-the-nuclear-launch-code-at-u...

> Those in the U.S. that had been fitted with the devices, such as ones in the Minuteman Silos, were installed under the close scrutiny of Robert McNamara, JFK's Secretary of Defence. However, The Strategic Air Command greatly resented McNamara's presence and almost as soon as he left, the code to launch the missile's, all 50 of them, was set to 00000000.

> Oh, and in case you actually did forget the code, it was handily written down on a checklist handed out to the soldiers.

Re:Summary of the Amazon S3 Service Disruption by timdorr on Thursday March 02, 2017 @01:54PM (#13777007)
I think you have it backwards. The post does not say they will simply be training the problem away. They are putting safeguards into their tooling to prevent the case of a fat finger.
Re:Summary of the Amazon S3 Service Disruption by baq on Thursday March 02, 2017 @01:54PM (#13777011)
The article leaves little doubt that they didn't know such an event would be so hard to recover from. They knew it wouldn't be easy, but they were surprised by how bad it was.