However, you still have things like "this update severs the machine from the management system due to unexpected XYZ", "this update is fine until it's rolled out to 80% of the world, at which point interactions with the other deployed systems hammer the system so hard the management interface can't get in properly", "this update looked fine because it was using almost entirely cached data but once the caches all expired it turned out to be a disaster, now restoring is a nightmare because we had to roll back the version, empty the cache, and regenerate everything", and all the other edge cases that no matter what you do, will cause cascading failures at a huge scale.
No matter what rule set you write, something's going to get past it.
Or, to put it another way, if you aren't yet on the Pareto frontier between power and safety, sure, by all means go get your free safety and power. But you will hit a limit on the two before you have all the power and all the safety, and the limit you will hit is going to be uncomfortable in at least one direction.
Somethings probably randomly fail, but because entropy and time is randomly selected we find ourselves in the successful ones.
With 20.000 servers, for example, an automated process could roll out updates to 1,000 every hour during 20 hours, check that response times, system load, etc. are within limits for the updated set during each hour, and send out alerts and pause updating when they do not.
So, the user still would press one button to do the update, but the change would slowly take effect, allowing both the system and humans to take action if needed.
Main problem there is to keep things flexible enough to allow somewhat out of the box updates. And of course, that requires that you can run with half your servers on a different version of your software.
you probably also will have to forget doing the entire update in a single transaction.