Post-mortem analysis on Google App Engine outage

Re:Post-mortem analysis on Google App Engine outage by jacquesm on Thursday July 09, 2009 @03:50AM (#695434)

this is a puzzling bit:

"8:00 AM --- The cause of the GFS Master failures has not yet been identified. However, a similar-looking issue that had been seen in a different data center the week prior had been resolved by an upgrade to a newer version of the GFS software. This upgrade was already planned for the App Engine primary data center later in the week, so the GFS SRE decides to commence the upgrade immediately in an attempt to alleviate the problem."

So, they let their old version of the GFS continue to run in spite of knowing that there was a critical bug, and then only when it crashed they decided to TRY the upgrade to see if it would cure the problem ?

If that upgrade would have been done earlier they could have shaved off an hour of their outage. I'm sure it's easy to second-guess from the sidelines but this really does puzzle me.

Also in the 'what did we do wrong' section they completely gloss over this point.

[ Parent | on HN ]

Re:Post-mortem analysis on Google App Engine outage by nostrademons on Thursday July 09, 2009 @04:21AM (#695472)

[Disclaimer: I work at Google, but my project is about as far away from App Engine and GFS as you could possibly get, so on this issue I have no more information than a layperson.]

It's possible that nobody knew or thought that it was a critical bug. Oftentimes, you'll see a little bug somewhere, think "Oh, we should fix that eventually", but the conditions that would make it into a really big bug are rare or unknown. It's only when those conditions actually happen that you think, "Well shit. I guess we should've fixed that last week."

If a bug isn't believed to be critical, then waiting to deploy the fix absolutely is the right decision. There're all sorts of things that can go wrong with an upgrade, and doing it on a schedule lets you go through a rigorous QA process and monitor the push as it happens.

[ Parent | on HN ]

Re:Post-mortem analysis on Google App Engine outage by sriramk on Thursday July 09, 2009 @04:18AM (#695468)

[Disclaimer - I work on Windows Azure which could be considered a competitor to GAE]

You always have critical bugs in large systems. Unless it is causing an outage or impacting some critical scenario in a major way, you don't want to mess up your usual process. You want to do orderly upgrades to all your environments/clusters/DCs and short-cut/hot patch only in dire emergencies.

Also, it is hard to judge the severity of bugs, especially in large scale distributed systems.

[ Parent | on HN ]

Re:Post-mortem analysis on Google App Engine outage by boundlessdreamz on Thursday July 09, 2009 @04:03AM (#695444)

All upgrades should be rolled out gradually so that if the the upgrade triggers a bug or in general have some issues, all data centers are not affected at the same time.

[ Parent | on HN ]

Re:Post-mortem analysis on Google App Engine outage by rgoddard on Thursday July 09, 2009 @09:33AM (#695806)

Plus the update was already planned on being installed later in the week, so it was not like they were ignoring it.

[ Parent | on HN ]

Re:Post-mortem analysis on Google App Engine outage by brown9-2 on Thursday July 09, 2009 @06:44AM (#695584)

I don't think its clear that the query of death bug was included/fixed in the 8am GFS upgrade - they're just saying that an issue with similar external symptoms was recently fixed by upgrading to this newer version.

Sounds like the fact that this newer version was available was completely unrelated to this specific bug.

[ Parent | on HN ]

Re:Post-mortem analysis on Google App Engine outage by jacquesm on Thursday July 09, 2009 @06:53AM (#695596)

Yes, I understood that from reading the article, the point is that they still wasted an hour in trying to see if that would fix it or not because the symptoms were similar (crash of the master GFS server).

[ Parent | on HN ]