News for hackers. Stuff that matters.
Re:Today’s outage for several Google services by pfg on Friday January 24, 2014 @10:03PM (#7119661)
I'm sure they use Canary Deployments, Gradual Rollouts and what have you to update their services.

I suppose this is a hard problem to solve on a configuration change level though. Imagine the configuration change that triggered the bug was something like "hey load balancers, stop sending traffic to the cluster with that new version of service X which seems to cause elevated error rates." You don't really want that kind of change to take too long to propagate.

There are no comments on this comment yet.