Besides, hiding the how does contribute a tiny little bit towards the forgetting. Not much, but still. So we keep asciinema[1] recordings of the tool performing some of the important tasks, so they can serve as a quick look-up, while the code serves as a full-detailed look-up.
----
All operations during development are carried out manually, unless already supported by automation (and until, of course). When automating the new operations, a sysadmin must also conduct the steps manually, and also take care of considerations a developer might not have been aware of.
> What I didn't see mentioned is that when the automation fails for whatever reason, typically it is a five-alarm situation where minutes count.
Generally, this is easily-handled by redundancy, HA, and snapshot rollbacks. Of course anything that's a potential five-alarm situation has redundancy, right? :P
No, that's not right. When automation fails, it's typically not in five-alarm situations. During five-alarm situations, knowing how to do things manually won't help, because if you could do them manually, the automation would already work.
Until you have to 10 years later for whatever reason, and suddenly, you forget. And suddenly you have to look things up.