News for hackers. Stuff that matters.
Re:Amazon Web Services are down by yuvadam on Thursday April 21, 2011 @04:52AM (#2469840)
Current status: bad things are happening in the North Virginia datacenter.

EC2, EBS and RDS are all down on US-east-1.

Edit: Heroku, Foursquare, Quora and Reddit are all experiencing subsequent issues.

Re:Amazon Web Services are down by guynamedloren on Thursday April 21, 2011 @06:08AM (#2470006)
Indeed they are. Right before the issues began, I pushed a bad update to one of my Heroku apps, causing it to crash. A minute later I fixed the bug, re-pushed the git repo to Heroku... and nothing. I've been stuck with an error message on my website for hours. Unfortunate timing!
Re:Amazon Web Services are down by jrnkntl on Thursday April 21, 2011 @06:49AM (#2470113)
Staging servers are an easy thing on Heroku :)
Re:Amazon Web Services are down by tlholaday on Thursday April 21, 2011 @11:13AM (#2470916)
You have my sincere sympathy.
Re:Amazon Web Services are down by samratjp on Thursday April 21, 2011 @06:07AM (#2469999)
http://status.heroku.com/ for those on Heroku
Re:Amazon Web Services are down by mike-cardwell on Thursday April 21, 2011 @07:06AM (#2470149)
Is it just me, or is their status page down?
Re:Amazon Web Services are down by bskari on Thursday April 21, 2011 @10:43AM (#2470770)
I know you probably have your answer by now, but this site always helps me out when I have that question:

http://www.downforeveryoneorjustme.com

Re:Amazon Web Services are down by guynamedloren on Thursday April 21, 2011 @07:53AM (#2470266)
It's been on-and-off for the past few hours.. It's up right now (for me, anyway).
Re:Amazon Web Services are down by amorphid on Thursday April 21, 2011 @09:13AM (#2470473)
This morning from 5a - 6a Pacific time I was able to access my Heroku app just fine.
Re:Amazon Web Services are down by JonnieCache on Thursday April 21, 2011 @09:43AM (#2470562)
I can access all my heroku apps that have their own DNS. Anything with a .heroku.com subdomain is down for me. Frustrating, knowing that the apps are still running but aren't routable.
Re:Amazon Web Services are down by cooldeal on Thursday April 21, 2011 @09:55AM (#2470611)
Yay, cloud.
Re:Amazon Web Services are down by JonnieCache on Thursday April 21, 2011 @10:18AM (#2470677)
Please do not make content-free posts such as this. It adds no value to the conversation and is only noise.

If you actually wish to make a useful point about the practicality or otherwise of massively virtualised systems for webapp deployment, please do. It's going to take more than two words though.

Re:Amazon Web Services are down by cooldeal on Thursday April 21, 2011 @11:00AM (#2470845)
I guess you haven't seen the Microsoft ads about the cloud? http://www.youtube.com/watch?v=Lel3swo4RMc

Anyway, my bad, I was just trying to make a joke to lighten up the mood. Sorry.

Re:Amazon Web Services are down by JonnieCache on Thursday April 21, 2011 @12:55PM (#2471329)
You're right, I hadn't. Fair enough.
Re:Amazon Web Services are down by recoiledsnake on Thursday April 21, 2011 @11:06AM (#2470866)
I laughed, it's relevant and puts things in perspective if you had seen the ads. So it's not really content free even if it's just two words.
Re:Amazon Web Services are down by mtogo on Thursday April 21, 2011 @10:35AM (#2470748)
Please do not make content-free posts such as this. It adds no value to the conversation and is only noise.

If you actually wish to bitch about a post that we all got the point of, you're going to need more than two paragraphs.

Re:Amazon Web Services are down by antonioe on Thursday April 21, 2011 @08:22AM (#2470340)
Foursquare is up now. Quora not showing the 503 anymore, Reddit still down.
Re:Amazon Web Services are down by jedberg on Thursday April 21, 2011 @09:36AM (#2470538)
We rely heavily on EBS still, so this is hurting us more than most others. Hopefully they'll have us back up soon.
Re:Amazon Web Services are down by bvi on Thursday April 21, 2011 @10:52AM (#2470808)
You guys might have answered this in one of your AMAs/blog posts (or was it raldi who commented?), but what options can reddit resort to should this stuff happen again to this degree of severity?
Re:Amazon Web Services are down by jedberg on Thursday April 21, 2011 @11:07AM (#2470870)
We're moving away from the EBS product altogether. The hard part is dealing with the master databases. Normally I'd have a master database with a built in raid-10, but I can't do that on EC2, so I have to come up with another option.

So I guess that is the long way of saying that hopefully it won't happen again.

Re:Amazon Web Services are down by jjm on Thursday April 21, 2011 @01:55PM (#2471575)
I do not believe you could be effective by moving away from EBS, you know without giving up quite a bit.

Doing things the right way with EC2 means using EBS. It's the brake caliper to the rotor. Sure you could have drum brakes but they're not nearly anywhere effective as they quickly get heat soaked. I'm referring to S3.

One should trust ephemeral storage. Your instance can go down at anytime. Write speeds to S3 are not nearly as fast as ephemeral or EBS arrays (raid).

Hate to say it, but If one cannot trust EBS then what the heck are 'we' doing on EC2... EBS quality should be priority one, otherwise we're all building Skyscrapers on foam foundations of candy cane rebar.

Re:Amazon Web Services are down by joshhart on Thursday April 21, 2011 @03:44PM (#2472065)
I can't say whether much has changed within the last year, but when I worked at FathomDB we had serious issues with EBS. You couldn't trust it. Odd things would happen like disks getting stuck in a reattaching state for days and disks having poor performance.
Re:Amazon Web Services are down by epi0Bauqu on Thursday April 21, 2011 @11:58AM (#2471093)
How do you move away from EBS and still deal with large data?
Re:Amazon Web Services are down by obfuscate on Thursday April 21, 2011 @04:15PM (#2472184)
Not sure what you had in mind by "large", but instance storage goes up to 1.7TB: http://aws.amazon.com/ec2/instance-types/
Re:Amazon Web Services are down by Devilboy on Thursday April 21, 2011 @08:25PM (#2472893)
The reason Reddit uses RAID10 is for performance, not disk size. A single instance storage device is just too slow for the Reddit database.
Re:Amazon Web Services are down by obfuscate on Friday April 22, 2011 @03:31AM (#2473495)
Many instance types have 2 or 4 virtual disks (presumably on different physical disks).
Re:Amazon Web Services are down by joshhart on Thursday April 21, 2011 @01:22PM (#2471438)
I imagine they'd do consider some combination of the following (sorted by most likely)

1. Sharding data 2. Pulling tables out to other servers from the main DB 3. Pruning excessive data 4. Compressing data

Re:Amazon Web Services are down by jjm on Thursday April 21, 2011 @01:58PM (#2471587)
It still has to be stored somewhere though right? If it's EBS you've just made yourself a complicated solution that will eventually fail all over again. No?
Re:Amazon Web Services are down by joshhart on Thursday April 21, 2011 @03:42PM (#2472053)
If the data is sharded, then the data/server is small enough enough to fit within the individual server's disk and you no longer need EBS to store it.
Re:Amazon Web Services are down by shrike on Thursday April 21, 2011 @09:06PM (#2472973)
NOTE: I work for Gluster.

We have had a lot of success stabilizing EBS by creating mdadm arrays out of lots of smaller EBS volumes. There is minimal additional costs and you can get better performance, stability, and protection (RAID 5, 6).

Gluster makes an OSS distributed filesystem that runs across availability zones, our AMI (not OSS) builds multiple RAID arrays on each instance then spreads the filesystem across instances in multiple AZs. Send me an email if you want to chat.

Re:Amazon Web Services are down by oomkiller on Thursday April 21, 2011 @02:53PM (#2471834)
Please tell us how you plan on moving 700 EBS volumes to something completely different. It sounds amazing.
Re:Amazon Web Services are down by bad_user on Thursday April 21, 2011 @09:59AM (#2470620)
Foursquare is up, but Quora is still showing the 503.
Re:Amazon Web Services are down by antonioe on Thursday April 21, 2011 @09:36AM (#2470537)
I stand corrected. 4sq down again.
Re:Amazon Web Services are down by wmoxam on Thursday April 21, 2011 @06:11AM (#2470015)
Not all EC2 & EBS instances are down. I have several in US-east-1a and 1 is down, while all of the others are working.
Re:Amazon Web Services are down by jedberg on Thursday April 21, 2011 @06:13AM (#2470023)
Same with us. About 10% of our 700+ volumes are having problems right now.

It's hard to tell for sure since there isn't any load.

Re:Amazon Web Services are down by nostromo on Thursday April 21, 2011 @06:19AM (#2470040)
We're fine on EC2 -- but everything on RDS seems to be giving us big problems. We started a few backups before we new it was systemic, and all of them are stuck at 0%. We also tried spinning up new instances -- and they're all stuck in booting.
Re:Amazon Web Services are down by obfuscate on Thursday April 21, 2011 @04:14PM (#2472181)
I think which physical data center "us-east-1a" etc. corresponds to differs from user to user, to load-balance given that people will probably be more likely to use 1a than the other zones.
Re:Amazon Web Services are down by agotterer on Thursday April 21, 2011 @11:20AM (#2470943)
We had about 45 min of downtime around 4am EST. Our RDS instances, EBS backed and normal instances all returned without problems. We are in Virginia us-east-1a and us-east-1b.
Re:Amazon Web Services are down by potomak on Thursday April 21, 2011 @06:10AM (#2470011)
Also Cuorizini (http://cuorizini.heroku.com) is down!