Post-mortem for Partial Cloudflare Outage on October 25, 2022

Re:Post-mortem for Partial Cloudflare Outage on October 25, ... by hayst4ck on Wednesday October 26, 2022 @04:01AM (#33341014)

87 sounds large

   2022-10-25 17:03: CDN Release is rolled back in Atlanta and root cause is confirmed.
   2022-10-25 17:28: Peak impact with approximately 5% of all HTTP requests resulting in an error with status code 530.
   2022-10-25 17:38: An accelerated rollback continues with large data centers acting as Upper tier for many customers.
   2022-10-25 18:04: Rollback is complete in all Upper Tiers.
   2022-10-25 18:30: Rollback is complete.

But this seems like the 87 minutes referred to.

The 503 graph appears to show a steep drop between 17:38 and and 17:45, which looks like the "rollback" speed to me.

I think they likely drained many of the lower tier caches/pops/edges after which roll back speed doesn't matter very much since the machines being rolled back likely weren't servicing traffic.

> large data centers acting as Upper tier for many customers.

To me this sounds like a very awkwardly worded way to say "we shifted traffic".

Clarification on whether traffic was shifted or tiers were drained would be nice.

[ Parent | on HN ]

Re:Post-mortem for Partial Cloudflare Outage on October 25, ... by hayst4ck on Wednesday October 26, 2022 @05:20AM (#33341409)

Additionally, this was a code release based outage(?). It seems like a graph annotated with releases (or a/b testing changes) would have made this outage somewhat trivial to identify.

  2022-10-25 14:39: Multiple teams become involved in the investigation as more customers start reporting increases in errors.
  2022-10-25 17:03: CDN Release is rolled back in Atlanta and root cause is confirmed.

This is what I find worrying.

How does cloudflare track their code releases and does cloudflare annotate their 4 major graphs (errors, rps, utilization, latency) with lines showing when things were pushed to prod?

[ Parent | on HN ]

Re:Post-mortem for Partial Cloudflare Outage on October 25, ... by yjftsjthsd-h on Wednesday October 26, 2022 @10:26AM (#33344200)

I dunno; they say

> At first glance, the errors looked like they were caused by a different system that had started a release some time before.

And with somewhat noisy error rates I'm not sure even annotating graphs with releases would have made it unambiguous.

[ Parent | on HN ]