That's the "commit-confirm" process they mention they will use in the write-up:

> Primarily, we will be concentrating on automation improvements ... and provide an automated “commit-confirm” rollback.

Surprised everyone has not switched to this already - great idea

I assume there's some non-trivial caveats when using this with a widely-distributed system.

That's what is suggested in the blogpost as one of future prevention plans.

There was a common pattern in use back in the day when I managed openbsd filewalls (can't remember if it was ipf or pf days). When changing firewall rules over ssh, you'd use a command line like:

$ apply new rules; sleep 10; apply original rules

If your ssh access was still working and various sites were still up during that 10sec you were probably good to go - or at least you hadn't shut yourself out.

Back when I was a briefly a network engineer at the start of my career, on cisco equipment we'd do 'reload in 5' before big changes - so it'd auto restart after 5 minutes unless cancelled.

I'm sure there were and are better ways of doing it, but it was simple enough and worked for us.

most ISP tier routers have an entire commit engine to load and apply configs.

junipers allows for instance, one to do the command commit confirmed, which will apply the configuration, and revert back to the previous version if one does not acknowledge this command within a predifined time. this prevents permanent lockout out of a system.