I also use k3d and k0s:

https://k3d.io/ https://github.com/k0sproject/k0s

I wonder how big of an effort it would be for Apple to take some engineers and make Mac OS support kernel namespaces/cgroups that would bridge the gap between all of these cool container features Linux has that requires a virtual machine to recreate on Mac.

They should enable nested virtualization on M1/M2 first. As it stands now it's impossible to do anything with KVM without dual booting Asahi

Wait really? How does Docker macOS work on M1/M2? Or does it not?

Docker Desktop runs dockerd in a Linux VM with Apple's hypervisor framework. You can also run containers in a Linux VM with Parallels or VMware Fusion hypervisors, QEMU on macOS also uses HVF not KVM. But you can't run VMs inside those VMs as it stands today. This works fine on Intel Macs which means you can't experiment and use KVM - one of the killer features of Linux and things like https://kubevirt.io/ and firecracker. Or VMs running stuff like Proxmox TrueNAS or ESXi (all possible on Intel macs and every x86_64 CPU)

It seems to me that the right fix is for Docker Desktop to support M1. Docker, kubelet, the k8s control plane, and everything else has supported ARM for ages. There is no need for that extra VM and therefore no blocker on nested virtualization.

It's not an ARM problem, it's a kernel/OS one. Same as Windows, macOS simply doesn't have what it takes (namespaces, cgroups, etc.) to run Docker/Linux containers natively, therefore an intermediary Linux VM is needed.

Doh, I forgot about that. You're right, the VM is needed anyway.

Agreed, but considering that kubernetes now supports joining windows workers to run windows containers, as well as integrated support for dockerd inside wsl2... That leaves macOS as honestly the worst platform for any kind of container related work.

> Docker Desktop runs dockerd in a Linux VM with Apple's hypervisor framework

Which in my experience uses at a minimum 4gb of RAM just sitting idle with no containers running

Would this solve anything? Most containers use a linux userspace, which requires an actual linux kernel.

Your idea would more similar to Flatpak (containerized GUI apps).