Re:Post-mortem analysis on Google App Engine outage
by nostrademons on Thursday July 09, 2009 @04:21AM
(#695472)
[Disclaimer: I work at Google, but my project is about as far away from App Engine and GFS as you could possibly get, so on this issue I have no more information than a layperson.]
It's possible that nobody knew or thought that it was a critical bug. Oftentimes, you'll see a little bug somewhere, think "Oh, we should fix that eventually", but the conditions that would make it into a really big bug are rare or unknown. It's only when those conditions actually happen that you think, "Well shit. I guess we should've fixed that last week."
If a bug isn't believed to be critical, then waiting to deploy the fix absolutely is the right decision. There're all sorts of things that can go wrong with an upgrade, and doing it on a schedule lets you go through a rigorous QA process and monitor the push as it happens.
It's possible that nobody knew or thought that it was a critical bug. Oftentimes, you'll see a little bug somewhere, think "Oh, we should fix that eventually", but the conditions that would make it into a really big bug are rare or unknown. It's only when those conditions actually happen that you think, "Well shit. I guess we should've fixed that last week."
If a bug isn't believed to be critical, then waiting to deploy the fix absolutely is the right decision. There're all sorts of things that can go wrong with an upgrade, and doing it on a schedule lets you go through a rigorous QA process and monitor the push as it happens.