Re:Post-mortem analysis on Google App Engine outage
by sriramk on Thursday July 09, 2009 @04:18AM
(#695468)
[Disclaimer - I work on Windows Azure which could be considered a competitor to GAE]
You always have critical bugs in large systems. Unless it is causing an outage or impacting some critical scenario in a major way, you don't want to mess up your usual process. You want to do orderly upgrades to all your environments/clusters/DCs and short-cut/hot patch only in dire emergencies.
Also, it is hard to judge the severity of bugs, especially in large scale distributed systems.
You always have critical bugs in large systems. Unless it is causing an outage or impacting some critical scenario in a major way, you don't want to mess up your usual process. You want to do orderly upgrades to all your environments/clusters/DCs and short-cut/hot patch only in dire emergencies.
Also, it is hard to judge the severity of bugs, especially in large scale distributed systems.