You ended up with lots of magic an internal domain knowledge about how to do things, and often had to get clever to get the result you wanted. They were often somewhere both not opinionated and too opinionated, and generally suffered from being "DSL"s domain-specific languages which were really just syntatic magic in various languages which really needed to be actual programming languages or more explicitly policy setting frameworks. They were in the middle of several extremes and tried to be lots of things at once and generally just didn't do a great job, unfortunately.

How does Kubernetes not also result in the same? I'm not questioning plenty of improvements and the somewhat different domain it brings to the table overall but genuinely curious. From what I've seen any tool designed to integrate and manage complex, disparate systems is going to end up with lots special cases, domain knowledge, and require digging into implementation details on a regular basis for anything other than absolute common-case uses.

Here's the difference between kubernetes and ansible (and alikes) (and I'm over-simplifying a bit of course, but the idea is there)

* With kubernetes, you write down the state you want your infrastructure to be in, and k8s figures out how to get there

* With ansible, you write down how to get to the state you want and ansible runs those steps, and hopefully they end up in the same state. The issue is that if you run the same set of steps on a Debian linux distribution, and on a Ubuntu distribution, well, they might end up in a similar state, but they actually aren't; for starters: one is debian, the other is unbuntu.

    - name: "Ensure sshd is started and enabled on boot"
      systemd:
          state: started
          enabled: yes
          name: sshd

But this is not perfect, because it's easy to eventually end up with imperative commands, like "run this script", or "Copy this file".

>With kubernetes, you write down the state you want your infrastructure to be in, and k8s figures out how to get there

My guess is that this devolves to "run this script" or "copy this file" in practice as well. Or it just becomes tautological, i.e. "here's the state I want the system to be in (and that means run this script and then copy this file over here, and don't forget to symlink it over there)". Am I wrong?

What's happened though is that Kubernetes expects for you to hand it an artifact that encapsulates all of the messy steps required before it can "run the thing." In practice that means the "run the script and symlink stuff" kind of glue is still present, it's just moved into Dockerfiles (which are, more or less, just fancy shell scripts themselves), build scripts, CI/CD pipelines, and startup scripts.

The big shift is more that the responsibility for managing this junk is pushed, organizationally, onto whoever knows how to operate the actual application (because they'll be creating the Dockerfile). Usually that's a developer.

So it's not wrong to say:

> With kubernetes, you write down the state you want your infrastructure to be in, and k8s figures out how to get there

But it's not really a complete explanation, because it assumes you've already gotten to the point that you've bundled an artifact up that will "do the right thing" when Kubernetes provides the fundamental pieces of infrastructure you define in your template.

Terraform isn't the best implementation of it; k8s has full control over the infra so it's even more powerful in that regard.

Now whether its the right abstraction is a matter of debate. But ultimately like VMs or containers, its so far proven to be a valuable tool, at least for medium sized companies.

lots of environment specific magic and flow-breaking with every upgrade of anything

same with cloud envs and terraform

The other benefit that k8s got for "free" was with containers: developers could stuff in whatever environment, software and configs they wanted inside the container. This removed most of the "my app broke because some shared lib didn't install properly" sorts of issues. This separation b/w software configuration and application configuration is extremely powerful.

The reason it's popular is that it's backed by Google and solves problems Docker the company couldn't.

That doesn't mean k8s is by any means not suffering the same problems of being a dsl that should be something else.

It also doesn't mean k8s won't suffer the same weirdness between platforms (AWS vs GCE vs GKE).

This is a good point, and most problems came because the tools were still only involved in mutating state of the machine. They eventually evolved to be declarative, but that's just cosmetic.

In the CM tools if you added declaration that package should be installed and you removed it the package will still be there. You had an entry to say that it should be uninstalled.

This basically ensures that machines that are configured the same way often ends up being drifted apart.

NixOS solves this problem by having language that instead of describing what should be updated instead has a declarative language that describes how the entire system should be built. When you change configuration it actually rebuilds the OS from scratch. It might seem like a lengthy process, but it is actually a quick because of caching. Nix just fetches missing pieces from repo and places in ints store, rebuilds things that are not in the binary cache and then updates symlinks to new locations. Because of using symlinks, upgrades are atomic, and you can also roll back your changes.

The catch? It is a paradigm shift, it also doesn't help that the language used is functional. So it's a steep learning curve.

Containers are only somewhat used for scalable microservices. They're mostly just an easier way to deploy software than creating your own RPM package and using orchestration tools to deploy it on VMs.

This assertion makes absolutely no sense at all, and reads like a Markov chain of buzzwords.

Kubernetes is containers autoscaling, blue-green deployments, revertible deployment histories.

Ansible and the like is configuration as code. That's it.

The closest there is between kubernetes and configuration-as-code tools is Docker's dockerfile.

  [business requirements] -> ... -> [deployed and working software]

Where they differ, is the approaches that they take to achieving that result:

  Ansible: [run a playbook ahead of time] -> [the environment for your app will be set up] -> [you can deploy to it through Jenkins/GitLab/whatever] -> [your app will run, but Ansible will know almost nothing about it unless run]
  Kubernetes: [create a cluster ahead of time] -> [you'll be able to run any containers on it] -> [you can deploy to a registry through Jenkins/GitLab/whatever] -> [Kubernetes will make sure that your app remains running]

If you want fault tolerance, you have to do additional work with Ansible, like writing systemd services for restarts, manually setting up your load balancers and working on ensuring proper failover, as well as service discovery. And i'd reason that if something is hard to do, oftentimes it simply won't be done at all! Yet with Kubernetes, a lot of that comes out of the box (even though i think that it's often too complicated, K3s does seem sane, but personally Docker Swarm or Hashicorp Nomad hit the sweet spot of features vs complexity).

(Yeah, k8s does a whole lot more and is more opinionated, but still...)

And of course you’re right, the use cases don’t totally overlap. However there is a big chunk of the core functionality which is absolutely competing ways to solve the same set of problems.

It's clear when you deal with both types of systems. Perl optimized for stability and backwards compatibility at the cost of new features and new language conveniences. This is great and a strength when all your competitors are playing catch-up or don't have enough market share to make an impact, but once that's not the case, anything new they bring to the table is a point in their favor, and you get what we have now, the long slow death of Perl.

Backwards compatibility is wonderful until there's a really useful feature that it prevents, then all of a sudden it's a more complex story because it's not just preventing busywork, it's preventing progress.

It's a similar attitude to the parts of the OSS community who feel that anyone complaining about poor documentation or usability just "doesn't want to do the work" to understand the system.

I was contrary to the idea of 3rd party software repositories, and was of the opinion that one should strive to depend as much as possible from system provided packages.

But the same experience that you mention now, is what gave my point of view a 180 degree change, and now I'd rather install something from pip or NPM. I can just pin the desired version, and they more or less have the policy of holding past versions in a frozen state, so if I know for a fact that version 1.2.3 is exactly right for my needs, I'll keep using that one regardless of system.

The weak point of packaging systems (my experience is Debian and Ubuntu) is that normally there are not many (if _any_ at all) alternative versions offered for the same software. They are built with the general assumption that you should use the latest ones, and of course they don't even attempt to offer a history of versions of every package. So every time you upgrade, it happens what you say, "divert attention to the tool's problems instead of using the tool to solve problems" (very good wording by the way, I liked it a lot)

https://semver.org

Semver doesn't make life easier. Making backward compatible changes is what makes life easier.

at this point i've started transitioning as much of my administration stuff as i can away from ansible. there's some good stuff in their codebase, but it's easier for me to find it in the source code and copy-paste it into my own projects than it is for me to work within the ansible structure. ansible solved a lot of problems for us, but over the time we've been using it i'm pretty sure it's caused more problems than it's solved.

Any preferred alternatives you can share?

The way Ansible is developed it’s way easier to fork your own and backport bug fixes and features you need because it’s a toss up whether the fix you need actually got merged or whether it languishes in the GH issues.

a combination of docker containers and terraform (for managing aws) is doing most of the heavy lifting for me.

Eventually I switched to Ansible because it made the more complicated things a lot simpler.

I do not think it is a case of hindsight 20/20. These are basic things like object need multiple fields but it is rigidly defined with 2 fixed strings and now one need object V2 to update. Or all functionality dumped at base URL so no scope of adding sub-functionality without breaking the whole pattern. I can understand no one can plan for how complex functionality evolve over time. But here I see perennial amateur hour with every release. And it is doubly irritating when these frameworks are already many layer deep stack of bullshit abstractions so ideally they should be able to handle changes without breaking.

Ansible at least had the advantage of being a 'broad' tool from the get go. So a lot of the common use cases were covered. But I'm sure surprises still popped up :-)

"Due to a limitation in pip, if you are upgrading from Ansible 3 (or earlier), you need to uninstall Ansible and Ansible Base before installing Ansible 4"

Python does some things really well, but their package managers leave what to be desired [especially for the uninitiated].

I don't even think backwards incompatibilities are the biggest reason. In practice, the problem is often around bugfixes and upgrades. Upgrades are much easier if you can do them independently. It sucks to have to upgrade the kernel and other tooling just to grab a small bugfix--or hold off on a transition waiting until both OS issues and code base issues are resolved.

> It's not exactly clear how can Perl programs or shell scripts or Makefiles from 20 years ago play perfectly fine unchanged

In all likelihood they don't. For open source projects they probably have a lot of workarounds and cruft to support a variety of environments and spans of versions they are both written against and tested against. Internal code is written and tested against a specific version relying on your proprietary environment.

this is not coupled to OS to begin with,ansible is just a bunch of python modules in a long trenchcoat

If this is the only issue, it sounds only mildly annoying. Either upgrade your tooling to use the newer version of be prepared to package up that old version of Puppet/Ansible--as long as you control what version is installed and are comfortable with the range of versions the package manager offers.

The type of scenario I had in mind is relying on something like Python, Ruby, Perl, etc. Something the OS may also deeply rely on. This came up a lot when macOS stopped updating a lot of GPL2 stuff to avoid shipping GPL3 or when they announced they'd stop shipping Python, Perl, and Ruby. You also see it when people try and change system Python version outside of the package manager in Linux and can no longer boot--the same thing can just as easily happen with libraries.

OTOH, in the case of Ansible, I've had good success with `pip install --user` to get a local (to my user) Ansible install that's separate from the system, and it looks like specific versions of software can be requested.