Re:Summary of the Amazon DynamoDB Service Disruption
by antirez on Wednesday September 23, 2015 @06:11AM
(#10264190)
This is a classic, even because those software parts usually are the least tested in a variety of conditions, even when they are relatively well tested. It's very hard to simulate real failures.
It seems that perhaps more time/effort needs to be spent testing the systems (and the use of those systems) which are critical for handling failure. While building them, it's easy to dismiss their scaling requirements on the basis that they shouldn't ever be under significant load.