In Telekube we have hybrid auto updates with fallback to manual in case if things go wrong, so we can recover:
How does this play out in an actual failure scenario? How can a administrator detect failures? Does it apply to the entire OS, including kernel upgrades, or just your own software?
Imagine etcd or docker fails during the upgrade on one of the master nodes.
Our monitoring system - satellite will report failure to the upgrade process that will stop.
Administrator will use our diagnostic utility to diagnose the problem, fix it and proceed with upgrade from the last failed step, but in the manual mode.
> Does it apply to the entire OS, including kernel upgrades, or just your own software?
Right now it only applies to our software
IMO good reliability practices counsel against this approach, certainly not without testing these upgrades in a staging environment first.