In the case of Talos, Kubernetes can provide the flexibility you want from a more traditional Linux distribution.

That sounds like Kool aid. Can you expound more on this?

With Kubernetes you can schedule workloads in a number of different ways. Let’s say you insisted on having a shell and package manager. Run your favorite distro’s container as a DaemonSet. With the proper mounts and permissions you can do a lot. In other words use Kubernetes to do the things need to do. Then what role does the OS really play? Well in the case of Talos it’s only there to run Kubernetes.

You literally do not need more than the most basic of host-OS facilities to run any Linux workload you want in a pod, the OS just needs to run containerd and have a kernel, talos adds the management plane to that mix to make it useable, but the userland provided by the oci image will not see any difference at all than if it were running on Ubuntu; it will see the kernel.

We use “system extensions” to give you flexibility while keeping the base small.

Want GPU drivers? Add the extension. Need Tailscale? Extension.

https://www.talos.dev/v1.6/talos-guides/configuration/system...

How is that different than adding binaries, in the context of this comment where the point was "In the end you end up just adding everything back in that you originally took out, because managing all the little weird different subsets is not worth the benefit."

In the context of that assertion, adding or subtracting "extensions" and adding or subtracting binaries are equivalent. Both are adding or subtracting "piece of code that provides fuction".

They are similar but extensions don't have to be binaries. It can also be files you need to be available before Kubernetes starts.

Talos is purpose built to run Kubernetes workloads and not general purpose Linux. Hopefully, you don't have to add _everything_ back to the OS, but we know some things cannot run as a container or Kubernetes workload.

Extensions are required for specialized hardware (eg network, GPUs) and is the closest thing to a "package manager" available in Talos. Extensions can be binaries but don't have to be. We have a lot of common extensions provided and maintained by us but anyone can create extensions as needed.

One nice thing about extensions is they get layered and you don't have to pre-build an artifact like you do with other Linux distros with something like packer. factory.talos.dev will let you pick you extensions and get an artifact no packer/bash/config management required.

Like the other commenter said, this is the kind of thing custom distros always have to do, that lead to either reinventing or just going back to a big fat distro. I've gone around this wheel a couple times. It's a nice idea to have something custom-tailored, but a tailored suit only fits one person, and is expensive.

You can still build a system to manage Kubernetes nodes without making your own distro. Even a heavily modified stock distro gains you benefits from basing off someone else's work. You can reuse the solutions they've made, and contribute back your customizations for your specific use cases.

That's how today's distro installers/package managers/etc came to include all the functionality they have. You couldn't eject a CDROM from a Busybox system, until one weird kid in high school decided to try to use Busybox to make a CD-bootable RAM-resident distro, found out it had no 'eject' command, and then sent a patch in to Busybox to add it. Now everyone can use that command, and that functionality is still there 20 years later.

It's also a lot harder for users to use proprietary solutions than ones they're familiar with. Your OS has no shell or console, only an API? So if there's a problem, how do I drop in with gdb, strace, tcpdump, and the entire suite of Linux debugging tools, to try and quickly diagnose and then patch an issue? I'm sure you've created some way to do it, but now I need to go find out how to do it, and probably use whatever stock tools are there, which may have their own quirks or incompatibilities.

But I get that a corporation's interest is mostly in "get something working now" as opposed to "get something working that will be better in the long run". DIY/NIH often becomes the engineering department's watchwords, and a custom distro is one of those eventualities.

We run many talos clusters at Civo, and they are far easier to manage than the other cluster types that use a standard Linux distribution stripped and stuffed with what we need, and the custom image build process is easy to get running in CI, all in all talos is wonderful both as tenant nodes and our region supercluster nodes, and much simpler process to add/remove nodes to the pool and do a few other k8s-centric tasks like etcd snapshot backups and pre-configuration of our regions before we have kit on the ground.

Executables also don't have to be binaries, and the executable interface is also an api.

A convincing argument might exist, but I haven't heard one so far.

MX Linux would be that "slim-yet-compatible" system. It can load itself completely into RAM, and has many distro-specific components such as Frugal Install and Remastering, yet also lets you install standard Debian packages.