Post-mortem for Partial Cloudflare Outage on October 25, 2022

Re:Post-mortem for Partial Cloudflare Outage on October 25, ... by oefrha on Wednesday October 26, 2022 @03:45AM (#33340910)

> And if your service is already down, you have nothing to lose and everything to gain.

They had a 5% request failure rate at the peak of this outage. So a botched rollback could definitely do more harm than good if the impact isn’t 100% understood.

[ Parent | on HN ]

Re:Post-mortem for Partial Cloudflare Outage on October 25, ... by londons_explore on Wednesday October 26, 2022 @04:19AM (#33341105)

But there is a good chance that was a 100% request failure rate for a specific service. And when you have 100% of requests failing for a service, you ought to be thinking about doing a fast rollback for that service as long as you have confidence that such a fast rollback won't take other infrastructure down.

[ Parent | on HN ]

Re:Post-mortem for Partial Cloudflare Outage on October 25, ... by jgrahamc on Wednesday October 26, 2022 @03:53AM (#33340971)

Exactly.

[ Parent | on HN ]