Re:Cloudflare outage on November 18, 2025 post mortem
by Feathercrown on Tuesday November 18, 2025 @10:58PM
(#45975702)
Incremental config changes sounds like it could lead to a LOT of bugs
Re:Cloudflare outage on November 18, 2025 post mortem
by vlovich123 on Wednesday November 19, 2025 @12:07AM
(#45976063)
Incremental in terms of 1% of the fleet using it, then 5% etc. this is standard course.
Another option is to make sure that config changes that fail to parse continue using the old config instead of resulting in an unusable service.
But the bigger change is to make sure that config changes roll out gradually instead of all at once. That’s the source of 99% of all widespread outages