Maybe some. Puppet, for example, fetches a manifest and applies it on the server. Failure to apply a manifest triggers re-running the manifest. In small installations it will be asking puppetmater(s). Large installation run masterless, where they fetch the repo and run it. I would be highly surprised if Ansible did not have this mode even if it is not a popular way of doing it.
> for i in {000..199}; do ssh machine$i "echo new_setting > /file"; done > for i in {000..199}; do ssh machine$i "grep new_setting > /file"; done > for i in {000..199}; do ssh machine$i "reload service"; done > for i in {000..199}; do ssh machine$i "verify reload"; done
You are running this sequentially and by hand?! What happens when the server is dead and you need a new one? You provision it by hand and go through the stages to get it into the state where it can accept the same cattle command?
> You are running this sequentially and by hand?! What happens when the server is dead and you need a new one? You provision it by hand and go through the stages to get it into the state where it can accept the same cattle command?
Depends on the change, but generally something like that. Change + Verify. It really doesn't take much more than 10-15 minutes for any given change. We've been at a stable state for about 2 years so changes are rare at this point, other than our security update cycle.
For rebuilds, we pull periodic disk images (monthly + any major changes) and can stand up a new server in about 20 minutes, almost all of which is hands-off.
After that it takes between 60 seconds and 5 minutes for it to be inserted into service regardless of the complexity of its configuration and registration.
I never take it on blind faith either:
for i in {000..199}; do ssh machine$i "echo new_setting > /file"; done
for i in {000..199}; do ssh machine$i "grep new_setting /file"; done
for i in {000..199}; do ssh machine$i "reload service"; done
for i in {000..199}; do ssh machine$i "verify reload"; done
As I was taught "never make a change without verifying it". All changes are documented as well.