Believe me, the last thing I want is to be up at 3am working on this. I'd much rather be sleeping and letting you not work.

Are you part of the team resolving it?

"Resolving" probably isn't the right word seeing as this is a purely amazon issue, not much they can do.

I'm guessing jedberg is mostly banging his head in walls and seriously looking at alternative hosting solutions right now.

Actaully, I'm on the couch in front of the fireplace, watching old SNL, waiting patiently for Amazon to fix their shit, and figuring out how we can not use EBS anymore.

I'm pretty sure redditors would be more than happy to deal with a day or so extra downtime as you guys switched to a better platform. Just leave a simple page up saying "Dumping Amazon, brb"... doubt you'd get many complaints.

TBH Amazon is so bad at this point that turning off Reddit is as good as trying to keep it running. Of course then you need to deal with the increased suicide rate.

Right on.

Periodically, latest a couple of days ago, there's a post / discussion about whether outsourcing core functionality is the right thing to do. There are valid points on both sides of the issue.

For my part, if I'm going to be up in the middle of the night I'd rather be up working on fixing something rather than up fretting and checking status. But either way things get fixed. The real difference comes in the following days and weeks. When core stuff is in the cloud then you can try to get assurances and such, fwiw. When core stuff is in-house then you spend time, energy and money making sure you can sleep at night.

A couple of years ago you had expressed interest in making a port to App Engine, any interest in doing that still? Want any help? ;)

I think it would take a lot more time than we have to make that work. Our code is open source if you want to give a proof of concept a go. ;)

On the off chance a port to app engine coalesces around this comment, count me in :)

So I was half right? Awesome!

I was looking at EC2 until this!

I thought you could cluster your instances across many regions and replicate blah blah blah and change your elastic ip addresses in instances like this?

Is this a case that it's not being utilised or does that system not work?

I appreciate you are busy right now so I'm not expecting a reply any time soon.

That is the theory, but all of our data is currently locked in the inaccessible EBS system.

I'd still say Amazon is a great place for your startup, just don't use EBS.

I thought you could snapshot drives across regions and bring those EBS drives up under new instances in new regions?

I have not used it all in detail yet so I don't know the practicality of this method.

I think I will stick to my co-location costings I am doing for the time being. There is only one person to rely on when it all goes wrong then!

Good luck getting it sorted, i know I wouldn't appreciate being up at 3am sorting it though

> I thought you could snapshot drives across regions and bring those EBS drives up under new instances in new regions?

In theory, yes. In practice, those snapshots hurt the volume so much that it is impossible to take one in production.

Interesting, your insight has given me a lot to think about.

Do you guys blog this anywhere?

Yeah, usually they are on our blog right after the downtime, or in /r/announcements on reddit.

Or have a fallback, maybe?

Yeah, I'm one of the reddit admins.