Isn't that inherent because Docker needs a Linux kernel, so running it on Darwin has a hard requirement on virtualization and running an entire guest operating system?

Virtualization and even running a guest kernel and OS isn't 5x expensive on the CPU, so the problem is not virtualization per se.

Perhaps the problem is Darwin.

Docker abstracts and re-implements a lot of the filesystem and IO stuff; almost all of the issues (often outright bugs) I've had with Docker on Linux are related to that, too.

I know it's not bad on CPU, but I thought it took more of a hit on I/O? Granted, I suspect that some of this is indeed some issue with Darwin itself, or at least poor integration with it.

I agree, I/O can be pretty expensive.

It depends quite a lot on the kind of I/O and how it's implemented, so design choices matter.

If there's a full Linux kernel inside the VM, then you may as well do I/O inside the VM as well, using something like virtio and a ringbuffer of async block device commands to the host, or at least batching them. That will be quicker than relaying every POSIX file operation synchronously to the host, because the number of VM exits is much lower in the former case.

That's MacOS' fault, in fairness. MacOS does not support containers or the Linux abi in general so you're forced to run docker containers in a Linux VM, with all the CPU and I/O penalties.

It’s MacOS’s fault for not being Linux?

Microsoft added native container support to Windows. Apple has chosen to essentially ignore them.

Microsoft is a much more developer focused company IMHO. Still kinda astonishing how many developers choose MacOS.

Correct me if I’m wrong but the Windows “native container support“ is only for Windows images right? Which no one uses, since 99% of popular software like Redis, Nginx, whatever are for Linux

I always wondered what percentage of apps in production are using Windows images compared to Linux.

There’s probably not a huge demand for native MacOS containers. Even MacOS VMs are a rarity.

MacOS VMs are only a rarity because you're only allowed to run them on Apple hardware. If that wasn't the case we'd likely even have cloud support for the platform and Cross-Platform development would be significantly easier.

Well, yes?

I’ve done way too much investigation into this. They are fixing these issues finally[0] (I’m mc0 mentioned in the thread). It does absolutely suck and is mostly caused by hacks to support macOS inotify event propagation into the docker VM. They are also working on making a different system for syncing data using Mutagen[1] that should fix this completely. That being said, it is very usable today depending on the workload.

0: https://github.com/docker/roadmap/issues/12#issuecomment-652...

1: https://github.com/docker/roadmap/issues/7

If it's a docker problem maybe you're running it with the vfs storage driver which copies every layer every time instead of using overlayfs? If not then it may be a problem with the virtualization solution, not docker itself. VM overhead shouldn't be 5x, not even for IO. Unless you're trying to mount parts of the host filesystem, that's slow with pretty much any virtualization solution, perhaps barring virtio-fs, but that's probably not supports on osx

I thought those were (partially?) addressed (or still ongoing?) with the use of bhyve/xhvye?