News for hackers. Stuff that matters.
Re:Post-mortem for Partial Cloudflare Outage on October 25, ... by stevewatson301 on Wednesday October 26, 2022 @02:59AM (#33340643)
The incident report is very light on details. For example, it mentions adding instrumentation caused the issue, but no details as to how why they'd need to remove headers or how they'd contain it in the future, except for that they'd need to "fail fast" in this situation.

(You can obviously catch it with a blue-green deployment, which they do mention, but at that point some small portion of traffic has already been affected.)

Re:Post-mortem for Partial Cloudflare Outage on October 25, ... by jgrahamc on Wednesday October 26, 2022 @03:42AM (#33340896)
This incident report was out within hours of the failure. The team is now doing an internal retro to look at decision speed, rollback speed, and the circumstances around that function having a side effect. We'll update the post with more detail.
Re:Post-mortem for Partial Cloudflare Outage on October 25, ... by cesarb on Wednesday October 26, 2022 @08:02AM (#33342416)
> We'll update the post with more detail.

Just a small suggestion: there's nothing in the post (unless I missed something) that mentions or even implies that this is not the "final report". You could have a small paragraph at the end saying "we might update this post further once we have more details", or even better, have a bullet point list of the things your team is still looking at. That way, readers of the blog post would know it might be worth it to come back later and read the post again.

Re:Post-mortem for Partial Cloudflare Outage on October 25, ... by stall84 on Wednesday October 26, 2022 @04:34AM (#33341194)
Yes they probably can fill in plenty of detail, but overall I was impressed with the incident report on it's own.. I don't think it's super common for companies' their size to release a timeline/report on the fkup unless they're coerced to.
Re:Post-mortem for Partial Cloudflare Outage on October 25, ... by dspillett on Wednesday October 26, 2022 @05:16AM (#33341384)
> For example, it mentions adding instrumentation caused the issue, but no details as to how why they'd need to remove headers

They probably don't need to at all, but I can think or several ways it might unintentionally happen.

Perhaps someone chose to implement the modification as “separate headers & body, modify headers, put the request back together” and something went wrong that made the putting-back-together fail quietly (so the now malformed request passes to the next part without an exception being raised) or not get called at all in some circumstances. Alternately a badly coded routine to remove a header, perhaps the instrumentation information after the point in the processes it is no longer useful, maybe removed more than it should.

How such things get past a code review without someone suggesting a less error-prone method might be a question that gets asked in the internal investigation. I assume somewhere like CF has many checks between initial code modifications and things dropping into production, unless they've really taken “fail fast” to heart!