Re:Post-mortem for Partial Cloudflare Outage on October 25, ...
by mritzmann on Wednesday October 26, 2022 @03:14AM
(#33340714)
> [..] and that the rollback lasted so long
That does not surprise me. There are companies working with CI/CD systems whose tests easily take more than an hour. And until they are completed, they cannot start releasing (enforced via system permissions). This is a good thing, but the possibility of an emergency rollback should be considered beforehand and maybe (it always depends) exceptions made for such purposes.
And sometimes it makes more sense to revert slowly to make sure it doesn't get worse.
That does not surprise me. There are companies working with CI/CD systems whose tests easily take more than an hour. And until they are completed, they cannot start releasing (enforced via system permissions). This is a good thing, but the possibility of an emergency rollback should be considered beforehand and maybe (it always depends) exceptions made for such purposes.
And sometimes it makes more sense to revert slowly to make sure it doesn't get worse.