News for hackers. Stuff that matters.
Re:Cloudflare outage on June 21, 2022 by testplzignore on Tuesday June 21, 2022 @09:00AM (#31823373)
Are there any steps that can be taken to test these types of changes in a non-production environment?
Re:Cloudflare outage on June 21, 2022 by vnkr on Tuesday June 21, 2022 @09:04AM (#31823429)
It's very difficult if not impossible to create a staging environment that would well enough replicate production at this scale. What bog posts suggest as a remediation in the process: "There are several opportunities in our automation suite that would mitigate some or all of the impact seen from this event. Primarily, we will be concentrating on automation improvements that enforce an improved stagger policy for rollouts of network configuration and provide an automated “commit-confirm” rollback. The former enhancement would have significantly lessened the overall impact, and the latter would have greatly reduced the Time-to-Resolve during the incident."