Summary of the Amazon S3 Service Disruption

Re:Summary of the Amazon S3 Service Disruption by djsumdog on Thursday March 02, 2017 @01:56PM (#13777037)

Which is really the point of automation and configuration management. When a manager asks you, "How are you going to prevent this in the future?" You can say, "We added a check so n must be less than x% of the total number of cluster members," or "We added additional unit tests for the missing area of coverage" or "We added new integration tests that will pick up on this."

Tests and configuration scripts don't prevent all breakage. But when you have them, you can say, "We missed that, let's add it," or "That failed, but it's a false positive. Let's add this edge case to this test."

If you have no automation, tests or auditing systems around running deployments, you can't do any of this.

[ Parent | on HN ]

Re:Summary of the Amazon S3 Service Disruption by jegoodwin3 on Thursday March 02, 2017 @04:19PM (#13778325)

I agree testing and automation are good. I think they need to go beyond this to formal verification, for something on this scale and reliability. NASA doesn't make these sorts of mistakes.

By the way - this is not just Amazon's problem now. We know the internet has a single point of failure. So does a lot of IoT.

When will we experience the first Suicide DevOps?

[ Parent | on HN ]

Re:Summary of the Amazon S3 Service Disruption by gmac on Thursday March 02, 2017 @04:28PM (#13778399)

NASA doesn't make these sorts of mistakes

https://www.wired.com/2010/11/1110mars-climate-observer-repo...

[ Parent | on HN ]

Re:Summary of the Amazon S3 Service Disruption by lisper on Thursday March 02, 2017 @05:26PM (#13778859)

https://www.youtube.com/watch?v=6OalIW1yL-k

(Specifically https://www.youtube.com/watch?v=6OalIW1yL-k#t=3m but it's worth watching the whole clip (or even the whole movie) if you haven't seen it before. It's from Terry Gilliam's "Brazil".)

[ Parent | on HN ]

Re:Summary of the Amazon S3 Service Disruption by icebraining on Thursday March 02, 2017 @06:00PM (#13779138)

Almost twenty years ago, though.

[ Parent | on HN ]

Re:Summary of the Amazon S3 Service Disruption by pron on Friday March 03, 2017 @06:41AM (#13782215)

Well, they've had plenty of opportunities to learn from their mistakes; Amazon hasn't had this long.

[ Parent | on HN ]

Re:Summary of the Amazon S3 Service Disruption by halomru on Thursday March 02, 2017 @04:45PM (#13778531)

>We know the internet has a single point of failure.

It has? I have yet to see the day where I can neither reach my email provider nor Google nor Hackernews. My local provider might screw up occasionally, or some number of of websites go unreachable for whatever reason. But I fail to come up with anything short of cutting multiple see cables that causes more than 50% of servers to be unreachable to more than 50% of users.

[ Parent | on HN ]

Re:Summary of the Amazon S3 Service Disruption by jsudhams on Thursday March 02, 2017 @10:19PM (#13780498)

https://en.wikipedia.org/wiki/2008_submarine_cable_disruptio...

[ Parent | on HN ]

Re:Summary of the Amazon S3 Service Disruption by pron on Friday March 03, 2017 @06:40AM (#13782210)

Amazon do formally verify AWS (they use TLA+), which is probably why this failure is a human error. Of course, you could expand the formal analysis of the system to include all possible operator interactions, but you'll need to draw the line at some point. NASA certainly makes human errors that result in catastrophic failures. The Challenger disaster was also a result of human error to a large degree[1]; to quote Wikipedia: "The Rogers Commission found NASA's organizational culture and decision-making processes had been key contributing factors to the accident, with the agency violating its own safety rules."

[1]: https://en.wikipedia.org/wiki/Space_Shuttle_Challenger_disas...

[ Parent | on HN ]