Post-mortem analysis on Google App Engine outage

Re:Post-mortem analysis on Google App Engine outage by ggruschow on Thursday July 09, 2009 @06:03AM (#695543)

Am I reading this right? I hope not.

Another user of GFS in the same primary datacenter as App Engine is issuing a request to the GFS servers that reliably causes a crash.

A google employee had a bug in their GFS-using code which ended up crashing GFS for themselves and all GAE users?

The failover procedure [...] was not designed to handle failover during full unavailability for a long period (greater than three hours).

Wouldn't something like a fire or a significant code bug in GFS cause full unavailability for hours?

[ Parent | on HN ]

Re:Post-mortem analysis on Google App Engine outage by robk on Thursday July 09, 2009 @11:43AM (#696074)

Yes, this is a bit curious. GFS cells are generally restricted to particular eng groups depending on the need, so I assume this particular GFS cell was fairly restricted, but it's still somewhat worrisome that unrelated, non-production code could lead to an outage of this scale. I expect whoever wrote the offending code is very aware of the repercussions of their work this week :)

[ Parent | on HN ]