Re:Cloudflare outage on June 21, 2022
by testplzignore on Tuesday June 21, 2022 @09:00AM
(#31823373)
Are there any steps that can be taken to test these types of changes in a non-production environment?
[ Parent
| on HN ]
It's very difficult if not impossible to create a staging environment that would well enough replicate production at this scale. What bog posts suggest as a remediation in the process:
"There are several opportunities in our automation suite that would mitigate some or all of the impact seen from this event. Primarily, we will be concentrating on automation improvements that enforce an improved stagger policy for rollouts of network configuration and provide an automated “commit-confirm” rollback. The former enhancement would have significantly lessened the overall impact, and the latter would have greatly reduced the Time-to-Resolve during the incident."