So, nothing here prevents "forgetting how to manage a server" like the article describes. This is literally how automations generally works. You are doing something manual. You know how to do it manually. You automate it so you don't have to do it manually. You never do it manually again.

Until you have to 10 years later for whatever reason, and suddenly, you forget. And suddenly you have to look things up.

Nothing short of regularly repeated practice prevents "forgetting how to manage a server" like the article describes, but that's not the only pitfall TFA refers to.

Besides, hiding the how does contribute a tiny little bit towards the forgetting. Not much, but still. So we keep asciinema[1] recordings of the tool performing some of the important tasks, so they can serve as a quick look-up, while the code serves as a full-detailed look-up.

----

1: https://asciinema.org

Which is why I think there should still be some manually-managed servers remaining (possibly something like testing). What I didn't see mentioned is that when the automation fails for whatever reason, typically it is a five-alarm situation where minutes count. Having to search through man pages or stack exchange is wasting valuable time during one of the few situations where time is super-critical.

> Which is why I think there should still be some manually-managed servers remaining (possibly something like testing).

All operations during development are carried out manually, unless already supported by automation (and until, of course). When automating the new operations, a sysadmin must also conduct the steps manually, and also take care of considerations a developer might not have been aware of.

> What I didn't see mentioned is that when the automation fails for whatever reason, typically it is a five-alarm situation where minutes count.

Generally, this is easily-handled by redundancy, HA, and snapshot rollbacks. Of course anything that's a potential five-alarm situation has redundancy, right? :P

> What I didn't see mentioned is that when the automation fails for whatever reason, typically it is a five-alarm situation where minutes count.

No, that's not right. When automation fails, it's typically not in five-alarm situations. During five-alarm situations, knowing how to do things manually won't help, because if you could do them manually, the automation would already work.

Could you clarify who “we” are and what the tools you use are?

"We" are my teammates and I at a small company in Mumbai, India.

The tools we use for this are just: a programming language (Python, looking to add some Hy), a rich SSH library in the programming language, and some wrappers we wrote around simple tools and techs (e.g., rsync, tls, git, sqitch etc.).

The wrappers themselves are (currently) 1261 lines of code (according to sloccount).

We arrived at this after building our automation in Ansible, and then Salt, and learning from all the troubles and frictions we went through with either. They're both fine tools themselves, great for many tasks we'll never need to do, but some of their design decisions are at odds with what we needed.

I was going to chime in with something along the lines of:

...and this is why I like Ansible. easy to follow scripts that do all the leg-work, and is reproducible at the command line...

So I'd love to hear of some of the hurdles you have encountered. So I can avoid them in the future.

BTW, I do agree with you. I vastly prefer a 'CLI fall-back' level of control, and that's the only way to keep core skills relevant. Another example of this: Do it once manually, then use automation to replicate it 1000 times.

Interesting to hear Hy mentioned in a production context. What advantage do you expect to get, besides the cool factor?

> What advantage do you expect to get, besides the cool factor?

1. There are small bits of repetition in some of our steps that cannot be trivially abstracted in Python without sacrificing the simple aesthetic of imperative Python code, but they are perfect for macros.

2. One of the components involves quite a bit of interaction with the user, based on some data. The data itself changes infrequently, and the interaction code is generated from the data, but the generator isn't as pretty (due to the nature of the data) as it could be in a Lisp.

These are not existentially important to the tool, but we've had bugs because of both. Something about the ugliness makes people not look at it too closely. While the bugs were small and easily spotted by an attentive eye, it'd certainly be nicer if the likelihood of future creation of similar bugs could be reduced.

I like the principles that you outlined about balancing automation with repeatable human processes. But you lost me at writing your own config management system.

That seems at odds with rule number 3 because sysadmins must have fundamental understanding of the systems/network/OS, in-depth knowledge of your custom homegrown config manager and the language it's implemented in in order to debug.

Happy to hear it works for you though. Personally I'm willing to accept the shortcomings of an existing config management system like Ansible because it's a system that is already understood by a large percentage of sysadmins and the core program does not require any maintenance time/effort from my team.

This is precisely why I feared building our own for a while. So we persisted with Ansible (and then later with Salt), making our peace with the tool as we went along. Where a module was desired but did not exist, we built our own modules. Where a feature was desired, or desired differently, or a bug was found, we patched the tool (and upstreamed our patches). Where there seemed no way to do something the way we wanted to do, we worked around the tool, by calling out to other code, or asking to be called from other code.

Ultimately, it turned out:

1. We had already grown a custom configuration manager within the tool.

Both tools support many ways of doing the same thing, so we had to pick one way and constrain ourselves to it (e.g, master-minion vs. salt-ssh vs. masterless). This happened quite a bit. And, as usual, with enough use emerges a pattern. Plus, some ways simply did not exist and had to be built.

2. We had already learned large portions of the tool.

Ansible and Salt are simple tools when used for simple tasks. When using either for not-so-simple tasks, one invariably meets portions of their code/behaviour one doesn't expect to meet.

3. Any sysadmin we'd hire would need to know the configuration tool in-depth, anyway.

And, to our surprise, the vast majority knew only the basics, if that. Since we are, on principle, opposed to using something in an important capacity without understanding it well enough, and we needed to be certain any sysadmin we'd entrust with the responsibility did know their tool in-depth, we learned the tools in-depth, ourselves.

4. The tools are NOT simple.

When we'd learned Ansible and Salt, ourselves, we'd found they were actually quite a bit complex. Made sense, they had to take care of so many conditions and variations and different situations.

5. Any sysadmin we'd hire would need to know programming and a programming language.

We already had extensions in these tools, custom modules written in a real programming language. And, in this day and age, anyone with a responsibility as important as managing our production servers has to be able to program anyway.

----

Our current ops tool is only 1.2k (library) + 1.3k (config mgmt, sans static) LOC, the config mgmt is in plain Python, and is vastly simpler compared to knowing how Ansible or Salt work or how to write Ansible or Salt modules. The in-depth knowledge required is much smaller too (since we don't have to take care of all the many ways Ansible and Salt could be used or the platform differences Ansible and Salt need to worry about); just Python, SSH, SFTP, Rsync, Git, Sqitch, TLS, and the OS we have chosen, almost all of which one needs to know anyway.

I do this with my ansible deployments at work. I first spin up a test VM or env, then manually configure what I am going to do and get it working properly. Then once that is done, I just export my bash history/whatever config files I was working on and rewrite the pertinent bits in ansible.

I presumed that everybody did this.

Sounds like you got burned by poor design choices and instead of fixing the problem just mandated manual fallbacks instead.

Seems kinda weak.

> ... just mandated manual fallbacks instead

I'm sorry I wasn't clearer in my original comment. We do not have manual fallbacks. Everything is automated, with the capability of going full manual if ever necessary, at any point. The rules are simply to ensure that this ability can always be relied upon, even if never needed.

At the least, automating the unknown (the first pitfall mentioned in TFA) is a dangerous "unknown unknown" risk, and the first rule tries to protect against that. Then there's the risk of being burned by "Unknown automation" (the second pitfall mentioned in TFA), which is why we have the second rule. And finally, simply pressing a button and being told everything has been done for you, without being shown how (or worse, even that) it was done, is just boring (not to mention how this supports the third pitfall mentioned in TFA).

> The rules are simply to ensure that this ability can always be relied upon, even if never needed.

"Have a pilot in the cockpit who can fly the plane even if you trust your autopilot very much"