> Once identified we expedited a rollback which completed in 87 minutes.

I appreciate that it's a massive infrastructure spanning the entire globe, but... 87 minutes to revert a change? I wouldn't want the job of fixing it, but that doesn't seem good enough when the impact is that bad.

So if any caches were cleared in the process of rolling back, a slow rollback makes perfect sense to me.

I would expect a company like cloudflare would have previous revisions of software on their machines that could be reverted to near instantly.

Another thing to consider is what capacity they run at and how long a service takes to initialize. If you can only take 10% of your machines out at any given moment and each one takes a while to initialize, you don't really have much choice in the matter.

1) I wonder if they so many layers of CI/CD and testing, that the whole pipeline, even rollbacks probably re-run their tests which takes time. Are CI/CD runners the problem or it is something like network/DNS thing?

2) What if someone inside Cloudflare glibly yells "Why can't we just rsync our code to all data centers in less than a minute?" What would be the pushback this person would get? Why is it a stupid suggestion? Answering that would reveal some crazy layers of abstractions that get piled up in large companies.

If their processes require a suite of tests be run on staging systems before the revert hits production, that may take considerable time depending on how much is potentially affected.

Also the 87 minutes might be for everything, perhaps it was a rolling update: update some nodes, test, update the next batch, repeat. This might mean that a good chunk of the infrastructure was updated for sooner and traffic could be diverted through those parts (presumably those nodes are over-specified, or just easily scaled, to be able to take a glut of extra load - as they would need to in more normal circumstances in order to contend with waxing & waning traffic patterns).

Perhaps it is even the case that the errant update didn't get out to all areas before the problem became apparent, so they were not rolling back over the whole network. Again this may offer the opportunity to divert load to non-affected areas while the others are rolled back.

So while the full revert took 87 minutes, things may have been up and running properly (except perhaps a little slower than usual) for most users/processes much quicker than that.

That does not surprise me. There are companies working with CI/CD systems whose tests easily take more than an hour. And until they are completed, they cannot start releasing (enforced via system permissions). This is a good thing, but the possibility of an emergency rollback should be considered beforehand and maybe (it always depends) exceptions made for such purposes.

And sometimes it makes more sense to revert slowly to make sure it doesn't get worse.