News for hackers. Stuff that matters.

Rackspace Outage

We're back up!

--------------------------------------

irc.freenode.com #slicehost - lets celebrate our down-time.

Alastair: We're currently experiencing a power outage in one of our datacenters, we're working as fast as possible to get the issue resolved and will update the status at status.rackspacecloud.com as soon as we have updates. I apologize for any inconvenience and thank you for your patience.

At least she didn't say "any inconvenience we may have caused you." Today is a rainy day in the cloud. :-'(

As of 12:35AM CST Rackspace Cloud engineers are seeing intermittent connectivity to our WC2 cluster in our Dallas - Fort Worth (DFW) and data center. We are working to resolve the issue as quickly as possible and will update the status post accordingly.

If you have any questions or concerns please contact our support via live chat or at 1-877-934-0407 international +1.210.581.040.

UPDATE: As of 1:15am CST, Rackspace Cloud engineers are still working to address the current connectivity issues. We are making significant progress and we will post another update here shortly.

UPDATE: As of 1:30am CST, service has been restored to our WC2 cluster. We are going to continue to monitor the situation closely. Additional updates to follow.

( 28 comments | 18 points | on HN )
Re:Rackspace Outage by jlm382 on Tuesday November 03, 2009 @02:37AM (#918840)
It's as if Rackspace has lost its key value proposition. The first time this happened, many of us thought "oh, it's a one time thing." But now that so many of us pay more money for something that goes down every few months, it's starting to get annoying.

In some ways, I actually feel sorry for the team at Rackspace. This hurts them big time.

Re:Rackspace Outage by timdorr on Tuesday November 03, 2009 @03:10AM (#918887)
It's certainly disheartening. These have all been power-related so far. Given the amount of money they should have spent on power systems, there's no reason this should happen. I've been to Savvis' facilities and they know exactly when a battery in their UPSes start to fail and how badly. Gensets get load tested regularly. Everything is fully redundant with instant failover. Yes, things can still go wrong, but they shouldn't be going wrong this often.
Re:Rackspace Outage by fuentesjr on Tuesday November 03, 2009 @02:43AM (#918848)
All sites are bound to go down sometime and sometimes
Re:Rackspace Outage by jlm382 on Tuesday November 03, 2009 @02:53AM (#918859)
Yes, but when 100% uptime and fanatical support are key to your branding, even a little downtime undermines the expectations of current and prospective customers.

They set a high bar for themselves. By saying that they have the lowest downtime in the industry, more eyes are looking at them to see that they do as they say. That's what makes this such a nasty situation.

Re:Rackspace Outage by smanek on Tuesday November 03, 2009 @03:05AM (#918879)
Just unfortunate this one was so soon after their last big outage (in July, iirc).
Re:Rackspace Outage by naz on Tuesday November 03, 2009 @02:51AM (#918858)
Slicehost had excellent service until they were acquired by Rackspace. Now they're on the second or third huge outage this year.
Re:Rackspace Outage by mahmud on Tuesday November 03, 2009 @02:56AM (#918862)
Rackspace copying Linode so hard they're even down-time compatible.
Re:Rackspace Outage by plinkplonk on Tuesday November 03, 2009 @02:58AM (#918865)
I have irate customers on the phone :-(

( I am not blaming slicehost but does "power outage" make any sense? Aren't there power backups? (and presumably an alert system for when primary power goes down?) If anyone has worked in/runs a hosting service, please enlighten me).

Re:Rackspace Outage by xenoterracide on Tuesday November 03, 2009 @03:07AM (#918881)
no a power outage never makes sense. all systems should be on online battery backup with gas powered backup generators for outages longer than 5 minutes. These systems should be tested periodically to make sure that they are all in good working order.

I think I'd rather not have 100% uptime and have periodically tested systems, to increase reliability as long as I know when my server MAY go down.

Re:Rackspace Outage by ErrantX on Tuesday November 03, 2009 @04:19AM (#918982)
There is a Rackspace update; from reading it this hit one of the fluke single points of failure in one of their power system.

Considering how data centres are set up there isn't much they could have done to stop it :)

Looks like human error (during the servicing)

Re:Rackspace Outage by tomjen2 on Tuesday November 03, 2009 @05:31AM (#919040)
Well I guess they need that fanatical support.

Personally I would just say fire the support people and use the money to hire somebody that can actually deal with servers.

Re:Rackspace Outage by ErrantX on Tuesday November 03, 2009 @06:39AM (#919085)
> Well I guess they need that fanatical support.

Im not a rackspace user :)

But yeh; someone is probably getting a tongue lashing. Human error is so easy; even for experienced engineers and especially with live electrical equipment... (been there, done that, got the scars)

Re:Rackspace Outage by xenoterracide on Saturday November 07, 2009 @03:55PM (#928299)
Reminds me of Liquid Web who copied rackspace calling it Heroic Support. If you need to be a Fanatic or hero you're doing it wrong. I think I'll start a web host and call it "competent" support.
Re:Rackspace Outage by nikcub on Tuesday November 03, 2009 @03:09AM (#918886)
TC was down - http://www.techcrunch.com/2009/11/02/large-scale-downtime-at...

although I was a little more gentle with them on the phone than @arrington (check his stream)

Re:Rackspace Outage by timdorr on Tuesday November 03, 2009 @03:16AM (#918895)
He's a bit...harsh is putting it lightly.

After the article on Facebook game scams, he was seeming to move past his previous attitude. Now he just looks like a raving mad man. It's not Rackspace's fault if you left a single point of failure on your site by hosting in one datacenter. If you want better reliability, then pony up the cash. You're making enough from just one of those ads to do it.

Re:Rackspace Outage by nikcub on Tuesday November 03, 2009 @03:20AM (#918903)
I don't think you grok sarcasm

(ok so let me explain. we are running with the joke that it was scoble himself that took the servers down.)

Re:Rackspace Outage by jollyjerry on Tuesday November 03, 2009 @02:41AM (#918845)
Ya my site's down too :( Statuses are being updated at:

http://twitter.com/#search?q=%23RackStatus

Re:Rackspace Outage by stevedewald on Tuesday November 03, 2009 @02:31AM (#918830)
All my sites are down too. I am not happy about this.
Re:Rackspace Outage by JimEngland on Tuesday November 03, 2009 @02:45AM (#918851)
We just invited a bunch of new users onto our site earlier tonight and I freaked out thinking they broke our server, until I saw this :)
Re:Rackspace Outage by oomkiller on Tuesday November 03, 2009 @03:47AM (#918938)
This is probably a much more complex failure than just a genset not working or dead UPS batteries.
Re:Rackspace Outage by ErrantX on Tuesday November 03, 2009 @03:03AM (#918873)
Everything breaks.

Personally it seems pretty good to get it fixed in an hour.

Re:Rackspace Outage by Rabidgremlin on Tuesday November 03, 2009 @03:03AM (#918876)
lol (to avoid crying) we just "launched" yesterday and we are down already: http://blog.split-the-bill.com/rackspace-cloud-is-down
Re:Rackspace Outage by mikeyur on Tuesday November 03, 2009 @03:05AM (#918878)
Did you just link to an entry that's hosted on the rackspace cloud and inaccessible? Just me?
Re:Rackspace Outage by Rabidgremlin on Tuesday November 03, 2009 @03:09AM (#918885)
Actually Posterous seemed to be up but they were not processing emails (perhaps that bit is the bit hosted in the cloud ?). I had to post via the web...

Ironic that we decided not to host our own blog so that we would have a working communications channel if our main site got taken out....

Re:Rackspace Outage by jonknee on Tuesday November 03, 2009 @02:41AM (#918847)
My slice managed to stay up... Phew. Good to see their being vocal about it.
Re:Rackspace Outage by tzury on Tuesday November 03, 2009 @03:35AM (#918920)
my slice power source is hybrid, so once the electricity went down, it switched to the solar module, unfortunately, it happened at night time, so this did not helped much
Re:Rackspace Outage by jnaut on Tuesday November 03, 2009 @02:25AM (#918824)
My demo is down too, hope this comes back soon. :(