News for hackers. Stuff that matters.
Re:Amazon Web Services are down by jedberg on Thursday April 21, 2011 @06:02AM (#2469978)
Thankfully, no. :)

But yeah, right now we're shutting everything down to try and avoid possible data corruption. Once they restore service, hopefully we'll be able to come back quickly.

Re:Amazon Web Services are down by pvh on Thursday April 21, 2011 @08:00AM (#2470285)
Hey Jedberg, if you guys aren't already rolling your own, check out fdr's WAL-E tool. It bounces postgres write-ahead logs off S3 and goes great with the new PG9 replication.

https://github.com/heroku/WAL-E

Re:Amazon Web Services are down by ghotli on Thursday April 21, 2011 @03:19PM (#2471956)
Thanks for this. I had designed and partially implemented this exact same thing. Do you know of this running in production anywhere?
Re:Amazon Web Services are down by stef25 on Thursday April 21, 2011 @08:01AM (#2470292)
Amazon is really not being kind to you guys; I sort of hope you'll find an alternative solution fast!
Re:Amazon Web Services are down by acangiano on Thursday April 21, 2011 @09:25AM (#2470505)
If I was Rackspace, I'd be at Reddit/Wired's headquarters already.
Re:Amazon Web Services are down by pbhjpbhj on Thursday April 21, 2011 @10:58AM (#2470831)
Didn't Jedburg say that they could reduce the failure by spending with Amazon.

I wonder if Rackspace really want this particular traffic burden. It seems that if Reddit choose not to pay for the load they need then you get lot's of bad press for it ... perhaps I'm seeing it wrong.

Rubbish analogy: Kinda like if I was doing a haulage business and you called out for a wheelbarrow to carry some elephants, then when the barrows broke we got bad press despite. If you'd paid for a heavy animal transport package ... OK it's all going wrong, you get the idea.

Re:Amazon Web Services are down by jedberg on Thursday April 21, 2011 @11:14AM (#2470918)
No, I said that we have spent all we can, and at this point we need development.

However, in this case, the outage is not because of any issues with our setup, but with Amazon.

Re:Amazon Web Services are down by pbhjpbhj on Thursday April 21, 2011 @06:05PM (#2472568)
>"we have spent all we can"

So is it a financial constraint with Amazon? Would you be suffering the same sorts of outages regardless of the technology on the backend or does AWS basically suck?

Re:Amazon Web Services are down by rm445 on Friday April 22, 2011 @02:11AM (#2473408)
You have the wrong end of the stick, because you're missing the history of the story. Reddit have a weird budget when it comes to staffing costs versus operating costs due to their parent company's policies as a media comapny - so they have a decent budget but are massively understaffed.

Statements like the one you're quoting are in that context. Let's say you have an unlimited operating budget - you can come up with all kinds of wonderful plans for massive redundancy and zero downtime. But you can't make that happen if you're not allowed to hire any engineers or sysadmins! As far as I'm aware reddit are paying Amazon mucho dinero but still having irredeemable problems with the storage product, EBS. They are stuck on an unreliable service without the manpower to move off.

That's the story, as far as I can piece together from comments here and on reddit.

Re:Amazon Web Services are down by pbhjpbhj on Friday April 22, 2011 @09:53AM (#2474037)
Ah, you see from what I read on reddit I understood that the staff shortage was simply part of Conde Nast's unwillingness to spend money on reddit and that constant downtime issues were another facet of that same problem.

It's not making money and those looking after reddit don't want to ruin it with a huge money grab - instead taking a soft approach, first just begging for money, then adding in a subscription model (freemium anyone?) and more subtle advertising by way of sponsored reddits (/r/yourCompany'sProduct type stuff).

I understand they've been hit with more staff problems just recently despite having a new [systems?] engineer start with them.

So in your view EBS is the problem regardless of finance? That was the nut I was attempting to crack. TBH I didn't expect someone at reddit to stick their neck out and say "yeah Amazon sucks" but they might have confirmed that the converse was true and they were simply lacking the necessary finance to support the massive userbase they have.

Re:Amazon Web Services are down by ulope on Thursday April 21, 2011 @05:50PM (#2472537)
Rackspace (and really all the "popular" US hosters) seem ridiculously expensive compared to hosting prices we have in Germany (see e.g. http://www.hetzner.de/en/hosting/produktmatrix/rootserver-pr... this is one of the biggest root server hosters in Germany).

Is this really so or are Racksoace and co. Just "boutique" offerings?

Re:Amazon Web Services are down by blhack on Thursday April 21, 2011 @12:45PM (#2471301)
Will you guys ever do something like this:

http://cache-scale.appspot.com/c/www.reddit.com/

but officially supported (and paid for) by you?

Re:Amazon Web Services are down by hysterix on Thursday April 21, 2011 @12:02PM (#2471117)
Sup jedberg, I obviously don't have nearly the level of knowledge with the intricacies of reddit, but coming from a strictly "business" standpoint, the amount of downtime reddit receives due to amazon issues is astounding. Perhaps it's time to look for alternatives?

Anyway, thanks for your time.

Re:Amazon Web Services are down by thirtysixred on Thursday April 21, 2011 @07:03AM (#2470143)
SHUT. DOWN. EVERYTHING.

Good plan though. :p

Re:Amazon Web Services are down by jedberg on Thursday April 21, 2011 @07:11AM (#2470165)
My original comment was "We're going Madagascar on the servers." Then I remembered I was on HN, not reddit. :)
Re:Amazon Web Services are down by JonnieCache on Thursday April 21, 2011 @10:42AM (#2470767)
>Then I remembered I was on HN, not reddit.

Right now we may as well be on reddit.

[deleted]
Re:Amazon Web Services are down by jedberg on Thursday April 21, 2011 @11:15AM (#2470926)
Don't do that. HN is not the place for memes.

reddit isn't either, but we lost that battle a while ago.

Re:Amazon Web Services are down by hysterix on Thursday April 21, 2011 @12:04PM (#2471127)
It's refreshing for me to see you say this. God speed soldier.
Re:Amazon Web Services are down by davidreiss666 on Thursday April 21, 2011 @10:22AM (#2470698)
You don't need to flee the country yet.
Re:Amazon Web Services are down by ILIKECAKE on Thursday April 21, 2011 @08:56AM (#2470421)
Hey man.. some of us would have got it :)
Re:Amazon Web Services are down by rmoriz on Thursday April 21, 2011 @07:10AM (#2470164)
President Madagascar is doin' cloud biz, too?
Re:Amazon Web Services are down by retube on Thursday April 21, 2011 @07:06AM (#2470151)
Reminds me of that scene in Jurassic Park
Re:Amazon Web Services are down by retube on Thursday April 21, 2011 @10:02AM (#2470632)
Good lord. What's with all the DVs??
Re:Amazon Web Services are down by retube on Thursday April 21, 2011 @11:09AM (#2470889)
Ha. HN really on it's high horse today.
Re:Amazon Web Services are down by eogas on Thursday April 21, 2011 @11:47AM (#2471046)
I know right? I made a hilarious joke a little while ago and jedberg yelled at me and everyone downvoted me. And I'll be very surprised if this comment doesn't get downvoted to hell too.

EDIT: I also simply greeted jedberg, and a bunch of people thought that was a good reason to downvote. Do people think there's an imminent influx of redditors, and that they have to dissuade them from becoming HNers? I don't think that's the case.

EDIT: Fuckin' called it.

[deleted]
Re:Amazon Web Services are down by jedberg on Thursday April 21, 2011 @11:12AM (#2470913)
I'm here all the time. :)