Post-mortem for Partial Cloudflare Outage on October 25, 2022

Re:Post-mortem for Partial Cloudflare Outage on October 25, ... by hayst4ck on Wednesday October 26, 2022 @05:20AM (#33341409)

Additionally, this was a code release based outage(?). It seems like a graph annotated with releases (or a/b testing changes) would have made this outage somewhat trivial to identify.

  2022-10-25 14:39: Multiple teams become involved in the investigation as more customers start reporting increases in errors.
  2022-10-25 17:03: CDN Release is rolled back in Atlanta and root cause is confirmed.

This is what I find worrying.

How does cloudflare track their code releases and does cloudflare annotate their 4 major graphs (errors, rps, utilization, latency) with lines showing when things were pushed to prod?

[ Parent | on HN ]

Re:Post-mortem for Partial Cloudflare Outage on October 25, ... by yjftsjthsd-h on Wednesday October 26, 2022 @10:26AM (#33344200)

I dunno; they say

> At first glance, the errors looked like they were caused by a different system that had started a release some time before.

And with somewhat noisy error rates I'm not sure even annotating graphs with releases would have made it unambiguous.

[ Parent | on HN ]