News for hackers. Stuff that matters.
Re:Post-mortem for Partial Cloudflare Outage on October 25, ... by londons_explore on Wednesday October 26, 2022 @03:27AM (#33340795)
> Once identified we expedited a rollback which completed in 87 minutes

Roll forwards and back should usually be done slowly and carefully.

However, during a major outage, there is a good reason to do the rollback very fast. Ie. Within 1 minute. Fast rollbacks end any outage sooner. And if your service is already down, you have nothing to lose and everything to gain.

That provides an interesting infrastructure problem. Fast rollbacks probably involve restarting a large number of servers, having a large number of cold caches, etc. You can find new bugs that only rear their heads during a fast rollback. Things like docker registries getting overloaded, kubernetes control plane being overloaded, other services becoming unhealthy due to a large influx of requests, etc.

The traditional approach is 'dont take risks, don't do a fast rollback'. But that can double or more the length of your downtime. Especially if you roll back the wrong thing first.

I'd therefore encourage preparing ahead of time for fast rollbacks. The dev/test environment should automatically do 'big bang' rollbacks every week. Production rollbacks should even be practiced from time to time, getting faster and faster each time, keeping an eye on metrics to look for instability.

Re:Post-mortem for Partial Cloudflare Outage on October 25, ... by avereveard on Wednesday October 26, 2022 @03:37AM (#33340856)
> The impact lasted for almost six hours in total.

And faster monitoring, they detected the issue after 4.5 hours and took one to fix

Re:Post-mortem for Partial Cloudflare Outage on October 25, ... by oefrha on Wednesday October 26, 2022 @03:45AM (#33340910)
> And if your service is already down, you have nothing to lose and everything to gain.

They had a 5% request failure rate at the peak of this outage. So a botched rollback could definitely do more harm than good if the impact isn’t 100% understood.

Re:Post-mortem for Partial Cloudflare Outage on October 25, ... by londons_explore on Wednesday October 26, 2022 @04:19AM (#33341105)
But there is a good chance that was a 100% request failure rate for a specific service. And when you have 100% of requests failing for a service, you ought to be thinking about doing a fast rollback for that service as long as you have confidence that such a fast rollback won't take other infrastructure down.
Re:Post-mortem for Partial Cloudflare Outage on October 25, ... by jgrahamc on Wednesday October 26, 2022 @03:53AM (#33340971)
Exactly.
Re:Post-mortem for Partial Cloudflare Outage on October 25, ... by quickthrower2 on Wednesday October 26, 2022 @05:31AM (#33341469)
Another chaos monkey case? Just do a fast rollback every day your D10 rolls a 1.
Re:Post-mortem for Partial Cloudflare Outage on October 25, ... by vitus on Wednesday October 26, 2022 @08:05AM (#33342442)
There have been YouTube outages that were exacerbated precisely because the SRE teams explicitly trained for the disaster scenario where everything is hard down and needs to be brought back as quickly as possible.