News for hackers. Stuff that matters.
Re:Summary of the Amazon DynamoDB Service Disruption by userbinator on Wednesday September 23, 2015 @05:41AM (#10264109)
Unavailable servers continued to retry requests for membership data, maintaining high load on the metadata service.

It sounds like the retries exacerbated the situation. I wonder if they use exponential backoff?

Re:Summary of the Amazon DynamoDB Service Disruption by BillinghamJ on Wednesday September 23, 2015 @06:01AM (#10264169)
It's amazing how frequently this seems to happen. A lot of the huge downtime events I've read about have occurred because of failures in the systems used to recover from failures.

It seems that perhaps more time/effort needs to be spent testing the systems (and the use of those systems) which are critical for handling failure. While building them, it's easy to dismiss their scaling requirements on the basis that they shouldn't ever be under significant load.

Re:Summary of the Amazon DynamoDB Service Disruption by antirez on Wednesday September 23, 2015 @06:11AM (#10264190)
This is a classic, even because those software parts usually are the least tested in a variety of conditions, even when they are relatively well tested. It's very hard to simulate real failures.
Re:Summary of the Amazon DynamoDB Service Disruption by bbrazil on Wednesday September 23, 2015 @06:05AM (#10264183)
> I wonder if they use exponential backoff?

More importantly, did they use randomised exponential backoff?

Having all the retires hitting at the same time can lead to a pulses of outages until things settle down.

Re:Summary of the Amazon DynamoDB Service Disruption by jmason23 on Wednesday September 23, 2015 @10:29AM (#10265166)
I have no doubt they would have been using randomised exponential backoff -- these issues are well recognised inside Amazon and best practices are well known. For example, here's a blog post from March this year from Marc Brooker on the topic: http://www.awsarchitectureblog.com/2015/03/backoff.html . It may not have been correctly tuned for this scenario however.

I'd say the Dynamo team are well aware of what they should have been doing, and kicking themselves for not foreseeing this cascading-failure case. ouch!

Re:Summary of the Amazon DynamoDB Service Disruption by allengeorge on Wednesday September 23, 2015 @08:41AM (#10264636)
Most exponential backoff implementations I've seen have an upper limit, where the initial (fast) retries work around transient networking blips.

My guess is that a case like this you'd hit the backoff's upper bound pretty quickly, and, given the large volume of servers hitting a comparatively small metadata pool, experience exactly the same failure mode.