News for hackers. Stuff that matters.
Re:Within 3 days another power outage at Linode (Fremont) by dholowiski on Tuesday November 23, 2010 @11:16AM (#1934145)
I worked in a datacenter (serving many companies you've heard of) during a catastrophic power failure that lasted almost 24 hours. It's kind of like a plane crash - it's never one thing failing that causes the problem (that's what redundancy is for) - it's that perfect chain of events, multiple 'once in a lifetime' failures that causes it.

For example, power outage occurs at the same time the UPS batteries are being changed. Bypass fails and Diesel generators fail to kick in. Circuit breakers blow everywhere making it extremely difficult to get the generators back on line. This all happens in the middle of the night in a winter storm (or 'lightning storm) which causes a further delay in response time.

Been there, done that.

Edit: Also, bureaucracy, lack of documentation, and a manager CF added several hours to the outage. Sometimes you just have to STFU and let the geeks fix the problem.

Re:Within 3 days another power outage at Linode (Fremont) by d2viant on Tuesday November 23, 2010 @11:49AM (#1934314)
Except, this isn't a once in a lifetime occurrence. As of today it's twice in a lifetime. That's indicates to me they didn't just hit that perfect chain of events, it tells me they have problems in their architecture or implementation that need to be addressed and tested before the next crisis occurs.

I worked in a datacenter as well. The whole point is that you test those scenarios when it's not an emergency so you can be confident everything will operate smoothly.

Re:Within 3 days another power outage at Linode (Fremont) by dholowiski on Tuesday November 23, 2010 @01:07PM (#1934676)
Don't discount the chance that this was caused by two totally different 'once in a lifetime' chain of events... i've seen it happen.

However, as you point out it is far more likely that there are architecture or implementation issues that caused the repeated outages.

There are so many reasons that I don't want to have servers in California... high costs, dense population vulnerability to earthquakes and a hot terrorism target. Massive bandwidth but massive utilization, somewhat shaky power grid, heck even the state has financial issues. I picked Atlanta for my Linode server which is better (it's still a little close to the coast for me) and I host my WordPress sites at a company in Dallas, TX. Did you know texas has their own power grid - separate from the rest of the USA?

Re:Within 3 days another power outage at Linode (Fremont) by jmcnevin on Tuesday November 23, 2010 @11:43AM (#1934267)
What you've described here would make an AWESOME movie. Granted, I'm not sure how one would work "Datacenter" into the title and still sell many tickets.
Re:Within 3 days another power outage at Linode (Fremont) by rryyan on Tuesday November 23, 2010 @03:50PM (#1935298)
Cory Doctorow (of Boing Boing fame) wrote a sci-fi novelette about a datacenter during a crisis: http://craphound.com/overclocked/Cory_Doctorow_-_Overclocked...

I personally didn't find this piece particularly great, but there's an entry in the genre for you.

Re:Within 3 days another power outage at Linode (Fremont) by dholowiski on Tuesday November 23, 2010 @11:55AM (#1934351)
What makes it even more exciting is that some of the companies were a bank and a hospital. Lesson learned - a bank gets far more anxious than a hospital when their servers go down.

I'm pretty sure there were Zombies roaming the aisles of servers where the power was out, that might help the box office sales.

Re:Within 3 days another power outage at Linode (Fremont) by tbrownaw on Tuesday November 23, 2010 @12:35PM (#1934541)
> multiple 'once in a lifetime' failures that causes it.

That reminds me of hearing mention of "multiple six-sigma events" I think it was, when things were crashing in 2008: the only sane conclusion is that the models are broken.