We are running our sites across three AZs, all in US-EAST. Our three web servers (one in each AZ) are not even able to load static test pages from their EBS volumes at this time. We also have a Percona MySQL cluster across three AZs and we are seeing very degraded perfomance in the EBS volumes on all three servers as well. It seems like this is impacting more than a single AZ from what I'm seeing.
Re:AWS issues, affecting Heroku and others
by blantonl on Monday October 22, 2012 @02:48PM
(#4684744)
Well, we've gone from "degraded performance" to completely losing our EBS instance store for our master MySQL server. If we don't see resolution shortly we might have to look at promoting one of our slaves and moving forward from there.
Re:AWS issues, affecting Heroku and others
by blantonl on Monday October 22, 2012 @08:03PM
(#4686179)
We ended up promoting a slave to master, and moving forward from there. Our master eventually returned to service, and we'll promote him tomorrow after things settle down.
Interesting to note that we issued a reboot on our Master and it went away and didn't return for over 45 minutes - we thought for sure it would have to be terminated. API calls and console access was severely restricted, so even launching in new AZ's was problematic.
Our Master MySQL server uses EBS (striped RAID across 4 EBS disks) and it is getting killed due to severely degraded EBS performance. There is definitely a major problem with EBS at this time.