News for hackers. Stuff that matters.
Re:Cloudflare outage on November 18, 2025 post mortem by shadowgovt on Wednesday November 19, 2025 @12:25AM (#45976166)
It looks like changing the permissions triggered creation of a new feature file, and it was ingestion of that file leading to blowing a size limit that crashed the systems.

The file should be versioned and rollout of new versions should be staged.

(There is definitely a trade-off; often times in the security critical path, you want to go as fast as possible because changes may be blocking a malicious actor. But if you move too fast, you break things. Here, they had a potential poison input in the pathway for synchronizing this state and Murphy's Law suggests it was going to break eventually, so the question becomes "How much damage can we tolerate when it does?")

Re:Cloudflare outage on November 18, 2025 post mortem by dwattttt on Wednesday November 19, 2025 @02:35AM (#45976827)
> It looks like changing the permissions triggered creation of a new feature file, and it was ingestion of that file leading to blowing a size limit that crashed the systems.

That feature file is generated every 5 minutes at all times; the change to permissions was rolled out gradually over the clickhouse cluster, and whether a bad version of that file was generated depended on whether the part of the cluster that had the bad permissions generated the file.