News for hackers. Stuff that matters.

AWS Elastic Block Store issues in one AZ in us-east-1

The Tarsnap website went offline at 1:05 PDT and attempts to launch a new instance in the same availability zone have failed. I'm also seeing the EC2 Management Console getting very slow -- possibly because lots of other people are logging in to investigate their failing systems.

As of 1:40 PDT I'm seeing EBS requests served and the Tarsnap website is back online.

Amazon's status page reports:

1:22 PM PDT We are investigating degraded performance for some volumes in a single AZ in the US-EAST-1 Region

1:29 PM PDT We are investigating degraded performance for some EBS volumes and elevated EBS-related API and EBS-backed instance launch errors in a single AZ in the US-EAST-1 Region.

RDS and ELB are also now (1:40 PM PDT) showing "investigating connectivity issues". Sounds like there was a core networking outage in that AZ.

2:21 PM PDT We have identified and fixed the root cause of the performance issue. EBS backed instance launches are now operating normally. Most previously impacted volumes are now operating normally and we will continue to work on instances and volumes that are still experiencing degraded performance.

( 28 comments | 51 points | on HN )
Re:AWS Elastic Block Store issues in one AZ in us-east-1 by alrs on Sunday August 25, 2013 @06:48PM (#6274078)
It used to be when EBS wiped out it didn't respect the boundary between availability zones. Every AZ in a region would be lost in tandem.

If they have managed to keep this incident confined to one AZ without trashing the whole region they have made significant progress.

Re:AWS Elastic Block Store issues in one AZ in us-east-1 by bgentry on Sunday August 25, 2013 @07:56PM (#6274305)
or maybe it just wasn't as bad as previous incidents :)
Re:AWS Elastic Block Store issues in one AZ in us-east-1 by jaytaylor on Sunday August 25, 2013 @04:54PM (#6273683)
This is the kind of thing you're susceptible to when you use AWS products that start with E: EBS, ELB, etc..
Re:AWS Elastic Block Store issues in one AZ in us-east-1 by nieksand on Sunday August 25, 2013 @05:18PM (#6273772)
At a minimum, you should use at least two availability zones if you care about high-availability on AWS. The vast majority of issues (including the current EBS hiccup) tend to be limited to one AZ.

For us, today's outage triggered a bunch of alerts, knocked 1/4 of our USE attribution servers offline (we're in four zones there) but otherwise caused no impact. Mostly just acked alerts and twiddled thumbs waiting for situation to resolve.

Re:AWS Elastic Block Store issues in one AZ in us-east-1 by gregr401 on Monday August 26, 2013 @01:11AM (#6275310)
No, this is the kind of thing that happens with a poor application architecture (ie, using one AZ). Sad that companies still can't get this right.
Re:AWS Elastic Block Store issues in one AZ in us-east-1 by mh- on Sunday August 25, 2013 @04:56PM (#6273689)
where's EC2 fall in that?
Re:AWS Elastic Block Store issues in one AZ in us-east-1 by weitzj on Sunday August 25, 2013 @05:20PM (#6273783)
if you want a safer EBS, create a raid ontop i and use snapshots.
Re:AWS Elastic Block Store issues in one AZ in us-east-1 by helper on Sunday August 25, 2013 @05:31PM (#6273806)
If you care about uptime you should really get off EBS (and services that depend on it like ELB and RDS). Having a network between your disks and your CPU make outages like this one inevitable.
Re:AWS Elastic Block Store issues in one AZ in us-east-1 by akurilin on Sunday August 25, 2013 @06:18PM (#6273976)
What do you recommend as an alternative for hosting large DBs on?
Re:AWS Elastic Block Store issues in one AZ in us-east-1 by insaneirish on Sunday August 25, 2013 @09:49PM (#6274698)
Re:AWS Elastic Block Store issues in one AZ in us-east-1 by akurilin on Monday August 26, 2013 @01:42AM (#6275382)
I can't easily get a feel for Joyent's pricing. Is it comparable/cheaper than AWS or?
Re:AWS Elastic Block Store issues in one AZ in us-east-1 by helper on Sunday August 25, 2013 @06:30PM (#6274026)
Use ephemeral disks on EC2 and a good distributed database like Cassandra or Riak. These scale well as your data grows and allow you to lose individual nodes without losing data (in sane configurations).
Re:AWS Elastic Block Store issues in one AZ in us-east-1 by eikenberry on Sunday August 25, 2013 @08:30PM (#6274428)
+1 to the use of Riak. It is great.

But if you are stuck using Postgresql or similar you put your live servers on Ephemeral instances and put one (unused otherwise) slave on EBS so you can snapshot it for backups. And you need at least 2 ephemeral systems so you have a slave to fail over to.

Re:AWS Elastic Block Store issues in one AZ in us-east-1 by joevandyk on Sunday August 25, 2013 @08:54PM (#6274509)
No need to use ebs/snapshots for backups.

Use wal-e and/or pg_receivexlog. Ideally run pg_receivexlog on a server at a different provider so that if amazon cancels your account your data is up-to-the-second recoverable.

Re:AWS Elastic Block Store issues in one AZ in us-east-1 by jacquesm on Sunday August 25, 2013 @06:25PM (#6274005)
Define 'large'. Up to 80T can be done in one box these days with up to 32 cores right next to it. Not cheap but fairly reliable. You'll still need to replicate off-site.
Re:AWS Elastic Block Store issues in one AZ in us-east-1 by akurilin on Sunday August 25, 2013 @06:41PM (#6274061)
So are you thinking along the lines of Hetzner?
Re:AWS Elastic Block Store issues in one AZ in us-east-1 by jacquesm on Monday August 26, 2013 @01:38AM (#6275372)
Sure. I've never used the cloud, I simply can't make the business case for it. Traffic or storage will kill it every time.
Re:AWS Elastic Block Store issues in one AZ in us-east-1 by aren55555 on Sunday August 25, 2013 @05:02PM (#6273718)
This is currently causing Heroku to have issues too. https://status.heroku.com/incidents/548
Re:AWS Elastic Block Store issues in one AZ in us-east-1 by jdleesmiller on Sunday August 25, 2013 @06:23PM (#6273996)
Now reporting as resolved... Except that my database is still down. Sigh. And it's not just me -- a few others have tweeted at @herokustatus.
Re:AWS Elastic Block Store issues in one AZ in us-east-1 by bgentry on Sunday August 25, 2013 @06:24PM (#6274002)
Hi, we usually track individual database issues via outbound support tickets opened on your account. The exception is for widespread outages, but this one was actually fairly small in scope.

If you didn't receive a support notification, please contact [email protected] so we can figure out why.

Re:AWS Elastic Block Store issues in one AZ in us-east-1 by jdleesmiller on Sunday August 25, 2013 @06:30PM (#6274027)
Thanks. I've now done this and referred some others to this comment.
Re:AWS Elastic Block Store issues in one AZ in us-east-1 by jday on Sunday August 25, 2013 @07:20PM (#6274201)
this affected me as well, i had to provision a new database and promote it to master in order to bring my app back online
Re:AWS Elastic Block Store issues in one AZ in us-east-1 by kennyma on Sunday August 25, 2013 @07:33PM (#6274236)
I had to do the same and Heroku support is non-responsive. This is beyond frustrating.
Re:AWS Elastic Block Store issues in one AZ in us-east-1 by josephpmay on Sunday August 25, 2013 @04:49PM (#6273661)
Is Instagram still on Amazon servers?
Re:AWS Elastic Block Store issues in one AZ in us-east-1 by flyt on Sunday August 25, 2013 @05:09PM (#6273735)
Yes.
Re:AWS Elastic Block Store issues in one AZ in us-east-1 by josephpmay on Sunday August 25, 2013 @05:12PM (#6273749)
I wonder if this outage will encourage them to migrate over to Facebook's servers.
Re:AWS Elastic Block Store issues in one AZ in us-east-1 by aioprisan on Sunday August 25, 2013 @06:17PM (#6273972)
doubt it, since they probably properly set up their infrastructure with multi-az setup
Re:AWS Elastic Block Store issues in one AZ in us-east-1 by nowarninglabel on Sunday August 25, 2013 @06:27PM (#6274015)
I was getting ngnix 504's on their site consistently during this outage time, perhaps coincidence, but seems likely to be related.