News for hackers. Stuff that matters.
Re:AWS outage summary by lukev on Friday October 26, 2012 @10:56PM (#4705180)
I am always astonished by how many layers these bugs actually have. It's easy to start out blaming AWS, but if anyone can realistically say they could have anticipated this type of issue at a system level, they're deluding themselves.
Re:AWS outage summary by bcantrill on Saturday October 27, 2012 @12:47AM (#4705387)
Full disclosure: I work for an AWS competitor.

While none of the specific AWS systemic failures may themselves be foreseeable, it is not true that issues of this nature cannot be anticipated: the architecture of their system (and in particular, their insistence on network storage for local data) allows for cascading failure modes in which single failures blossom to systemic ones. AWS is not the only entity to have made this mistake with respect to network storage in the cloud; I, too, was fooled.[1]

We have learned this lesson the hard way, many times over: local storage should be local, even in a distributed system. So while we cannot predict the specifics of the next EBS failure, we can say with absolute certainty that there will be a next failure -- and that it will be one in which the magnitude of the system failure is far greater than the initial failing component or subsystem. With respect to network storage in the cloud, the only way to win is not to play.

[1] http://joyent.com/blog/network-storage-in-the-cloud-deliciou...

Re:AWS outage summary by bigiain on Saturday October 27, 2012 @02:15AM (#4705536)
FWIW, once Amazon decided that an availability zone is an unreliable unit (and they tell you this up front, and strongly suggest running multiple AZ architectures for anything where you require reliability), then any cascading failure mode in a single AZ is not something you'd expect them to spend too much time protecting against. Sure, the cascade from EBS faults to RDS and ELB meant this affected more of their single AZ customers than it would have otherwise, but anyone using a singe AZ knew upfront that Amazon advised against that, and never claimed they intended to provide high-availability in single AZs.

So yeah, you're right - these systemic failures can be anticipated, and Amazon's advice (to spread your important infrastructure across multiple AZs) would have protected users from most of this. (I feel quite a lot of sympathy for the engineers involved in the cross-AZ failures this incident revealed - the multi-AZ RDS and ELB failures are things customers doing everything they were told got bitten by anyway, and are probably rightly annoyed...)

Re:AWS outage summary by seldo on Saturday October 27, 2012 @05:01AM (#4705775)
While Amazon do indeed say that multi-AZ is the way to go, their last 3 major incidents (including last year's cloudpocalypse) have all been full-region incidents.

IMHO, their biggest design problem is that they build their systems on top of each other (e.g. ELB is built on EBS and EIP). So when one system goes down, it takes down half a dozen others -- this is especially true of EBS, and especially dangerous because the services it takes down, like ELB, are the services people are supposed to be using to route around EBS failures.

Re:AWS outage summary by acdha on Saturday October 27, 2012 @10:00AM (#4706122)
They've had exactly one true region-wide failure - that 17 minute routing failure earlier this year. Running a truly multi-AZ setup has avoided every other outage popularly reported as “the cloud is falling”.

Some services - e.g. Heroku - have lots of impacted customers but that's due to their architecture, not the underlying AWS.

Re:AWS outage summary by seldo on Saturday October 27, 2012 @01:57PM (#4706737)
I'm sorry, but that's just not true. Prior to last week's incident:

June 28th 2012 - region-wide failure due to power outage and EBS dependency issues hitting ELB and RDS: http://aws.amazon.com/message/67457/

March 15th 2012 - 22-minute, region-wide networking interruption (no status report available)

April 2011 - cloudpocalypse, also a cascading EBS failure: http://aws.amazon.com/message/65648/

Re:AWS outage summary by acdha on Saturday October 27, 2012 @05:29PM (#4707305)
The only region-wide outage in your links is the network one I mentioned - they called it 22 minutes, I measured it as 17 on my systems. June 28th was indeed not region-wide - I have systems in every AZ and lost exactly one of them.

The other ones say “one of our Availability Zones” - which was rather my point: if you follow long-time accepted redundancy practices, you had far less - if any - downtime than people who put everything in one AZ or rely heavily on EBS volumes not failing.

Re:AWS outage summary by eclipticplane on Saturday October 27, 2012 @10:10AM (#4706134)
Mostly, but not entirely. Under the "Impact on Amazon Relational Database Service (RDS)" section they state that due to two other bugs, some Multi-zone RDS instances did NOT fail over, because their systems knew that the masters had stopped replaying to the standbys, but had continued to process some transactions (albeit more slowly, as the failures cascaded and amplified), and thus prevented the automatic promotion of a standby, as the standbys were now out of date.
Re:AWS outage summary by acdha on Saturday October 27, 2012 @05:31PM (#4707314)
I agree that's a problem but calling it a region-wide failure is an unhelpful exaggeration, particularly in the case of a SQL database where there's always a tradeoff between availability and consistency.
Re:AWS outage summary by antirez on Saturday October 27, 2012 @02:53AM (#4705604)
Very wise comment IMHO. More complex = more complex failure modes. The only way to improve is Less complex. Well also probably local storage has a number of other advantages from the point of view of performance.
Re:AWS outage summary by dchichkov on Saturday October 27, 2012 @05:34AM (#4705816)
I don't think that there is an obvious way to make it less complex. But complexity is not inherently bad. Internet is very complex, yet it is robust.

What feels wrong here, is that Amazon systems are too trusting to other Amazon internal systems.

Re:AWS outage summary by antirez on Saturday October 27, 2012 @08:59AM (#4706046)
Moving storage locally seems like a good move. When complexity is still too high and there are no ways to make it simpler (I don't think that's the case, btw) relax your requirements.
Re:AWS outage summary by digeridoo on Saturday October 27, 2012 @08:57AM (#4706042)
To be honest, EBS is the special sauce that makes me stick with AWS. There are so many little things that become so much easier by having network storage. To me the cost of upgrading or migrating machines using local storage is just not worth the perceived reliability gain to me.
Re:AWS outage summary by dmpk2k on Saturday October 27, 2012 @04:37PM (#4707164)
It's not just reliability though. The quality of I/O is also better. You don't see such slow I/O, with crazy fluctuations, with local storage.
Re:AWS outage summary by DanielRibeiro on Saturday October 27, 2012 @05:27AM (#4705808)
I believe the damage of cascading failures was also masterfully exposed during your talk on QCon Last year: http://www.infoq.com/presentations/Debugging-Production-Syst...
Re:AWS outage summary by pbhjpbhj on Saturday October 27, 2012 @06:22AM (#4705875)
Start at ~10:20 for the cascading failures.

Very entertaining IMO. But I would suggest the speaker cut the off-hand ostracism of large groups of people.