Re:AWS Service Disruption Post Mortem
by evangineer on Saturday April 30, 2011 @04:17PM
(#2501472)
They seem to have settled on a halfway house, pushing more of the control plane functionality down into the EBS clusters and making the remaining shared control plane more robust to the sort of failures that arose this time.
The API failures were ultimately tied to the network problems that occurred, not to a failure of the control plane.
EDIT: I should finish reading before I reply. :) It would appear that the network issue in the one availability zone was so severe that the control plane ran out of threads to service API requests to any of the availability zones.
So while it's true the underlying problem was a network issue, the fact that the the control plane is spread across availability zones was responsible for part of the outage that occurred across the whole region.
My totally unqualified assessment of this aspect of the outage is that, while it might make sense to have a control plane spread across availability zones, they presumably need to have isolated control planes for each zone, instead of a shared plane as they seemingly have now.