News for hackers. Stuff that matters.
Re:AWS outage summary by teraflop on Friday October 26, 2012 @11:15PM (#4705225)
> Multi Availability Zone (Multi-AZ), where two database instances are synchronously operated in two different Availability Zones.

> The second group of Multi-AZ instances did not failover automatically because the master database instances were disconnected from their standby for a brief time interval immediately before these master database instances’ volumes became stuck. Normally these events are simultaneous. Between the period of time the masters were disconnected from their standbys and the point where volumes became stuck, the masters continued to process transactions without being able to replicate to their standbys.

Can someone explain this? I thought the entire point of synchronous replication was that the master doesn't acknowledge that a transaction is committed until the data reaches the slave. That's how it's described in the RDS FAQ: http://aws.amazon.com/rds/faqs/#36

Re:AWS outage summary by mcpherrinm on Friday October 26, 2012 @11:21PM (#4705240)
This just sounds like a race condition gone weird.

I assume from this description that the update protocol looks something like this:

   process-request:
     Update-local
     push to sync buddy
     wait for sync buddy success
     reply-with-status
     mark state as synced
If the function does the local update and then gets stuck in the state waiting for the buddy to reply, one could imagine the failover daemon not handling that case very well. So while the master might not have acknowledged the transaction, the pair might get jammed trying to complete it.

EBS is a particularily complicated piece of software, and RDS is another layer of complication built on top of that. Bugs clearly happen, and it's an unfortunate state of affairs.

Re:AWS outage summary by redditmigrant on Saturday October 27, 2012 @02:55AM (#4705607)
I think aws uses drbd (http://www.drbd.org/) for replicating the writes to the standby. Among the replication modes of drbd suggested here: http://www.drbd.org/users-guide-emb/s-replication-protocols.... I suspect they probably use B and landed in a state where packets got delivered but werent written on the standby.