> You ended up with lots of magic an internal domain knowledge about how to do things, and often had to get clever to get the result you wanted.

How does Kubernetes not also result in the same? I'm not questioning plenty of improvements and the somewhat different domain it brings to the table overall but genuinely curious. From what I've seen any tool designed to integrate and manage complex, disparate systems is going to end up with lots special cases, domain knowledge, and require digging into implementation details on a regular basis for anything other than absolute common-case uses.

> How does Kubernetes not also result in the same?

Here's the difference between kubernetes and ansible (and alikes) (and I'm over-simplifying a bit of course, but the idea is there)

* With kubernetes, you write down the state you want your infrastructure to be in, and k8s figures out how to get there

* With ansible, you write down how to get to the state you want and ansible runs those steps, and hopefully they end up in the same state. The issue is that if you run the same set of steps on a Debian linux distribution, and on a Ubuntu distribution, well, they might end up in a similar state, but they actually aren't; for starters: one is debian, the other is unbuntu.

Ansible does try to do the 'write down the state you want your infrastructure to be in' thing. Eg,

    - name: "Ensure sshd is started and enabled on boot"
      systemd:
          state: started
          enabled: yes
          name: sshd

You're not saying 'enable the sshd service', but 'the desired state for sshd is started and enabled'.

But this is not perfect, because it's easy to eventually end up with imperative commands, like "run this script", or "Copy this file".

That does not seem avoidable. I have never used kubernetes, but this seems highly suspect:

>With kubernetes, you write down the state you want your infrastructure to be in, and k8s figures out how to get there

My guess is that this devolves to "run this script" or "copy this file" in practice as well. Or it just becomes tautological, i.e. "here's the state I want the system to be in (and that means run this script and then copy this file over here, and don't forget to symlink it over there)". Am I wrong?

Kubernetes itself tries Very Hard to completely abstract those implementation details. Like, obsessively hard. You (assuming "you" are somebody who wants to run something in k8s) should never need to know or care what it's doing "under the hood" to run your pod in the way you say.

What's happened though is that Kubernetes expects for you to hand it an artifact that encapsulates all of the messy steps required before it can "run the thing." In practice that means the "run the script and symlink stuff" kind of glue is still present, it's just moved into Dockerfiles (which are, more or less, just fancy shell scripts themselves), build scripts, CI/CD pipelines, and startup scripts.

The big shift is more that the responsibility for managing this junk is pushed, organizationally, onto whoever knows how to operate the actual application (because they'll be creating the Dockerfile). Usually that's a developer.

So it's not wrong to say:

> With kubernetes, you write down the state you want your infrastructure to be in, and k8s figures out how to get there

But it's not really a complete explanation, because it assumes you've already gotten to the point that you've bundled an artifact up that will "do the right thing" when Kubernetes provides the fundamental pieces of infrastructure you define in your template.

Yeah you're (mostly) incorrect. I recommend looking into terraform and how it handles state, that will give you a better idea of how it's possible to do infrastructure in a "this is what i want, figure out how to get there" way.

Terraform isn't the best implementation of it; k8s has full control over the infra so it's even more powerful in that regard.

Kubernetes doesn't attempt to be an API that can solve any orchestration issue, it is an abstraction on the underlying architecture. It's a lot easier to be backwards compatible here (and if you are not, create an operator that is).

Now whether its the right abstraction is a matter of debate. But ultimately like VMs or containers, its so far proven to be a valuable tool, at least for medium sized companies.

and then you use something like helm to orchestrate and end up being in the same place where you left ansible:

lots of environment specific magic and flow-breaking with every upgrade of anything

same with cloud envs and terraform

One of the best things about K8s is its pretty stable API. We use GKE and the upgrades happen without issues 99% of the time.

The other benefit that k8s got for "free" was with containers: developers could stuff in whatever environment, software and configs they wanted inside the container. This removed most of the "my app broke because some shared lib didn't install properly" sorts of issues. This separation b/w software configuration and application configuration is extremely powerful.

Yea, I think with both ansible (and stuff like puppet) and kubernetes they are both powerful tools that are pretty leaky abstractions. They are great for sharing and implementing recipes, but that abstraction layer makes it a little more difficult to become a master chef and understand/control exactly what's happening. They are awesome to have in your toolbox, but I still find it incredibly useful to have a tool like python in there as well.

I’m not saying kube really fixes all of these problems or is a better solution, but it exists as a response to a crappy solution. in other words one problematic solution replaced another problematic solution. Stay in the field long enough and you see a pendulum where one kind of solution with problems is replaced by a different kind of solution with different problems... and then it swings back and a new general goes back to a new iteration of the first kind of solution and gets to rediscover the old problems again, back and forth over and over.

Who said it doesn't? Look at its schema it's horrific.

The reason it's popular is that it's backed by Google and solves problems Docker the company couldn't.

That doesn't mean k8s is by any means not suffering the same problems of being a dsl that should be something else.

It also doesn't mean k8s won't suffer the same weirdness between platforms (AWS vs GCE vs GKE).

> The reason why kubernetes et al. are so popular is that the orchestration tools like ansible, puppet, salt, chef... they all failed to deliver ultimately great software. Plagued by mediocre quality external modules and a constant churn of semi-backwards incompatible changes which required frequent maintenance, they have just been hard to use.

This is a good point, and most problems came because the tools were still only involved in mutating state of the machine. They eventually evolved to be declarative, but that's just cosmetic.

In the CM tools if you added declaration that package should be installed and you removed it the package will still be there. You had an entry to say that it should be uninstalled.

This basically ensures that machines that are configured the same way often ends up being drifted apart.

NixOS solves this problem by having language that instead of describing what should be updated instead has a declarative language that describes how the entire system should be built. When you change configuration it actually rebuilds the OS from scratch. It might seem like a lengthy process, but it is actually a quick because of caching. Nix just fetches missing pieces from repo and places in ints store, rebuilds things that are not in the binary cache and then updates symlinks to new locations. Because of using symlinks, upgrades are atomic, and you can also roll back your changes.

The catch? It is a paradigm shift, it also doesn't help that the language used is functional. So it's a steep learning curve.

Nix is an incredibly far-sighted vision, and it's great to work with... once you get your head around it all. I much prefer to build containers using it, and it does a much better job of "provisioning" than any other tool I've used. I remember being appalled when I first used Ansible and discovering that adding a package, then removing the stanza to add that package, resulted in the package persisting on the system.

I've found that every new technology that solves problems, also comes with its own set of problems. :)

Ansible, chef etc. are provisioning tools, not container orchestrators.

The point is that had they worked well, containers may not have become as popular as they are now.

Containers are only somewhat used for scalable microservices. They're mostly just an easier way to deploy software than creating your own RPM package and using orchestration tools to deploy it on VMs.

I hope that you do understand the difference of server provisioning and container orchestration, as mentioned above, solve completely different problems.

Those things solve completely different problems.

> The reason why kubernetes et al. are so popular is that the orchestration tools like ansible, puppet, salt, chef... they all failed to deliver ultimately great software.

This assertion makes absolutely no sense at all, and reads like a Markov chain of buzzwords.

Kubernetes is containers autoscaling, blue-green deployments, revertible deployment histories.

Ansible and the like is configuration as code. That's it.

The closest there is between kubernetes and configuration-as-code tools is Docker's dockerfile.

I think it still makes some sense. For example, if you look at the process of software development like this:

  [business requirements] -> ... -> [deployed and working software]

Then it stands to reason that Ansible and Kubernetes are just two ways of getting to the same end result.

Where they differ, is the approaches that they take to achieving that result:

  Ansible: [run a playbook ahead of time] -> [the environment for your app will be set up] -> [you can deploy to it through Jenkins/GitLab/whatever] -> [your app will run, but Ansible will know almost nothing about it unless run]
  Kubernetes: [create a cluster ahead of time] -> [you'll be able to run any containers on it] -> [you can deploy to a registry through Jenkins/GitLab/whatever] -> [Kubernetes will make sure that your app remains running]

So essentially, i believe that many people have the stance, that they can probably just set up an environment for an application and forget about it (or at least not constantly monitor it), which is more in line with what Ansible provides, whereas Kubernetes is more suited to situations where it's impossible to have "stable software" (i.e. most business applications, since they don't have the quality of projects like the Linux kernel).

If you want fault tolerance, you have to do additional work with Ansible, like writing systemd services for restarts, manually setting up your load balancers and working on ensuring proper failover, as well as service discovery. And i'd reason that if something is hard to do, oftentimes it simply won't be done at all! Yet with Kubernetes, a lot of that comes out of the box (even though i think that it's often too complicated, K3s does seem sane, but personally Docker Swarm or Hashicorp Nomad hit the sweet spot of features vs complexity).

Ansible is way, way, way more popular than Kubernetes in the real world.

(Yeah, k8s does a whole lot more and is more opinionated, but still...)

Exactly : not everyone gets to play with a kube cluster. Also there is a lot of sysadmins that don't trust docker-in-prod...

It sounds like you think kubernetes et al. are a suitable replacement for ansible et al. , however k8s are not able to provision settings on a Web Application Firewall from F5 and the iLO links on your HP blade server farm. Et Al. Your statement works if you are letting AWS/MS/Google do all that for you and all you are responsible for are package dependencies and joining the right VLANs. Not everyone is so lucky and actual Configuration Management tools like Ansible+, while not perfect, make Domain Specific (i.e. real world Multi-Vendor) large environments run on time for most of the time.

I actually don’t like kibernetes.

And of course you’re right, the use cases don’t totally overlap. However there is a big chunk of the core functionality which is absolutely competing ways to solve the same set of problems.