Rackspace Outage
--------------------------------------
irc.freenode.com #slicehost - lets celebrate our down-time.
Alastair: We're currently experiencing a power outage in one of our datacenters, we're working as fast as possible to get the issue resolved and will update the status at status.rackspacecloud.com as soon as we have updates. I apologize for any inconvenience and thank you for your patience.
At least she didn't say "any inconvenience we may have caused you." Today is a rainy day in the cloud. :-'(
As of 12:35AM CST Rackspace Cloud engineers are seeing intermittent connectivity to our WC2 cluster in our Dallas - Fort Worth (DFW) and data center. We are working to resolve the issue as quickly as possible and will update the status post accordingly.
If you have any questions or concerns please contact our support via live chat or at 1-877-934-0407 international +1.210.581.040.
UPDATE: As of 1:15am CST, Rackspace Cloud engineers are still working to address the current connectivity issues. We are making significant progress and we will post another update here shortly.
UPDATE: As of 1:30am CST, service has been restored to our WC2 cluster. We are going to continue to monitor the situation closely. Additional updates to follow.
In some ways, I actually feel sorry for the team at Rackspace. This hurts them big time.
They set a high bar for themselves. By saying that they have the lowest downtime in the industry, more eyes are looking at them to see that they do as they say. That's what makes this such a nasty situation.
http://status.slicehost.com/2009/11/3/dfw-interruption-2
http://search.twitter.com/search?q=rackspace+OR+slicehost
( I am not blaming slicehost but does "power outage" make any sense? Aren't there power backups? (and presumably an alert system for when primary power goes down?) If anyone has worked in/runs a hosting service, please enlighten me).
I think I'd rather not have 100% uptime and have periodically tested systems, to increase reliability as long as I know when my server MAY go down.
Considering how data centres are set up there isn't much they could have done to stop it :)
Looks like human error (during the servicing)
Personally I would just say fire the support people and use the money to hire somebody that can actually deal with servers.
Im not a rackspace user :)
But yeh; someone is probably getting a tongue lashing. Human error is so easy; even for experienced engineers and especially with live electrical equipment... (been there, done that, got the scars)
although I was a little more gentle with them on the phone than @arrington (check his stream)
After the article on Facebook game scams, he was seeming to move past his previous attitude. Now he just looks like a raving mad man. It's not Rackspace's fault if you left a single point of failure on your site by hosting in one datacenter. If you want better reliability, then pony up the cash. You're making enough from just one of those ads to do it.
(ok so let me explain. we are running with the joke that it was scoble himself that took the servers down.)
http://twitter.com/#search?q=%23RackStatus
Personally it seems pretty good to get it fixed in an hour.
Ironic that we decided not to host our own blog so that we would have a working communications channel if our main site got taken out....