This is fine if you have a 32/64GB machine, but less so on an 8GB non-upgradeable laptop.
I get it - memory is relatively cheap these days - and manufacturers that are building memory-limited devices are really only doing it to fleece you on obscene upgrade fees at the time of purchase - but it would be nice if there was a more elegant solution to this on Windows and macOS.
WSL 1 had a solution to this that clearly took a lot of work to put together, wherein they'd have a Linux kernel running side-by-side as a Windows sub-process so that the memory pool was shared. Unfortunately it might have been too much work as they scrapped it entirely for WSL 2 and just used essentially the same VM route.
If anyone knows of any projects trying to work around that problem I'd love to hear about it. If Apple really wanted to bring the development community back on board, focusing on these kind of use cases would be great, sadly it seems someone over there has taken the view that scrapping butterfly keys and the touchbar is "enough".
Say what you will about Microsoft, but they've focused really hard on developer use cases for decades, and it shows.
When Windows containers are being used, it is just there to keep the docker daemon happy.
Containers are namespaced processes. These processes exec against the corresponding kernel they require. There is no workaround: if you have an ELF binary calling Linux syscalls it can only run on a Linux kernel†, so to run that you need a VM††. It's not as bad as it appears thanks to memory ballooning†††.
Conversely if you want to exec a Windows binary in a container, the Windows kernel needs to provide process namespacing features (which it does). And if you want to exec a Darwin binary in a container, then the Darwin kernel needs to provide process namespacing features (which it doesn't).
† WSL1 was implementing the Linux syscall API on the Windows kernel, which proved to be much more complex than it appears to be.
†† Or colinux (https://en.wikipedia.org/wiki/Cooperative_Linux), or user-mode Linux (https://en.wikipedia.org/wiki/User-mode_Linux).
FTFY ;) (and thanks!)
I think the only true solutions are (a) OS vendors develop their own native container platforms with UX similar or better than Docker (b) OS vendors agree on some common ABI standard
I've long wondered, and again now that Chat GPT is proving so adept at coding, if this translation layer could be automated. Do humans actually have to hand-code each syscall? Or are there just enough edge conditions that can't be automated?
So, I won't preclude the idea of AI helping, but I think human effort is still the bottleneck for projects like this. Even if AI could write perfect code 100% of the time, testing and troubleshooting would probably still be the larger timesink.
autoMemoryReclaim – Makes the WSL VM shrink in memory
as you use it by reclaiming cached memory
Obviously that limits the options, but I'll still be taking one last shot at using creative workarounds to tackle the memory problem in OrbStack (another containers-on-macOS product).
I imagine over time it will get smarter too. Right now it waits for no containers to be running for 30 seconds and enables resource saving mode but who knows what could happen in the future. Maybe it can internally profile and estimate load based on evaluating runtime stats of your contains and dynamically change the VM's resources on the fly and then expose a +% over provision threshold option or a way to turn off dynamic resource saver mode.
rund is an experimental containerd shim for running macOS containers on macOS.
rund doesn’t offer the usual level of container isolation that is achievable on other OSes due to limited macOS kernel API.
What rund provides:
- Filesystem isolation via chroot(2)
- Cleanup of container processes using process group
- OCI Runtime Specification compatibility (to the extent it is possible on macOS)
- Host-network mode only
- bind mountsExcept for bind mounts (not even overlayfs...) there isn't much interesting.
> - Host-network mode only
Yeah expect a lot of things to break in subtle ways... most containers are developed kinda expecting you have your own network namespace (and that no one else is using ports)
Most Linux (and Windows) containers. Since these are macOS containers there are no containers developed yet.
> So essentially a chroot with a bit of make-up
Well.
1. It is not trivial to properly set up a chroot on macOS. If you try to find a working guide/tool that works with modern macOS, I doubt you'll find anything (at least, I failed, even though tried very hard) 2. I believe that ability to package stuff into a Docker image distributable via already existing infrastructure and compatible with already existing tools maybe "a bit of make-up", but it is an important makeup. 3. Kubernetes recently got HostProcesses for Windows: https://kubernetes.io/blog/2022/12/13/windows-host-process-c.... They are even less isolated from host than chroot and still, people find them useful for certain scenarios.
> and a lot of marketing?
Thanks for "a lot of marketing", that made me chuckle. My own submission got buried yesterday with humble 8 points: https://news.ycombinator.com/item?id=37640688
Container definition is very stretched nowadays. Look at Windows HostProcesses in Kubernetes [1]. They don't have neither process, network nor device isolation from the host.
I also plan to try macOS sandbox-exec tool, which should offer additional isolation from the host.
[1]: https://kubernetes.io/blog/2022/12/13/windows-host-process-c...
I don't think so, but some Docker features could be implemented using XNU sandboxing AFAIK
> Furthermore, the existing container ecosystem assumes a Linux syscall interface. [1]. Does macOS provide that? I expect not.
This project is about running macOS containers on macOS. It's not about running Linux containers.
> I don't think so, but some Docker features could be implemented using XNU sandboxing AFAIK
Theoretically, probably, for coarse-grained yes/no things? I don't think it's able to go much further than "you can use the local network and/or internet" and "you can read/write to the filesystem location corresponding to you bundle identifier `com.foo.bar`" but not "hey let me present you with a namespaced view of loopback or process list".
Also not sure if it can be dynamically set by a parent process for a child? Seems like it's very bundle oriented (except maybe for Apple processes) so not very practical.
Yes, it can. See sandbox-exec tool. And I actually plan to use it: https://github.com/macOScontainers/rund/issues/15
I played with it some time ago, can't recall the context but it was about build systems / packaging (maybe nix?), doing the configure/make/make install with reduced privileges.
There is more to the container ecosystem than Linux containers; Windows native containers function much the same way (well, in two ways, with VM-backing or the traditional kernel syscall interface, but with Windows syscalls).
Essentially, like this:
https://ericchiang.github.io/post/containers-from-scratch/
https://earthly.dev/blog/chroot/
> The way Docker Desktop (and podman.io) implement "containers on macOS" is a bit of a cop-out
It's not, it's a requirement for running Linux containers: https://news.ycombinator.com/item?id=37656401
DfM is more like running the CLI locally to a remote Linux machine, and all it does is conveniently expose /Users in the same place through the VM folder share so that you have the convenient illusion that it happens locally.
If Darwin had process namespacing features it would not make it magically able to run Linux processes.
1. Rely on system call stability. This is like Linux containers but unlike Linux macOS doesn't provide a stable system call API. So this would break when the system updates with a change that modifies the system call API.
2. Install the host libraries into the container at runtime. This should provide as much stability as macOS apps usually have. It may also be beneficial as you wouldn't be embedding these into every container.
It seems like 2 would be preferable. However it may be a bit weird when building as the libraries you build against would be updated without the container being aware, but this is unlikely to break anything unless they are copying them to new paths which seems unlikely.
I'm really wondering, do you have any links about macOS syscall stability over versions?
* The most obvious is that commonly mentioned fact that syscalls may change. Here is an example where golang program broke because they were directly using the `gettimeofday()` syscalls[2].
* The interface between the kernel and the dynamic linker (which is required for since ABI stability for statically linked executables is not guaranteed) is private and may change between versions. That means if your chroot contains a `dyld` from an OS version that is not the same as the host kernel it may not work.
* The format of the dyld shared cache changes most releases, which means you can't just use the old dyld that matches the host kernel in your chroot because it may not work with the dyld shared library for the OS you are trying to run in the chroot.
* The system maintains a number of security policies around platform binaries, and those binaries are enumerated as part of the static trust cache[3]. Depending on what you are doing and what permissions it needs you may not be able to even run the system binaries from release of macOS on another release of macOS.
In practice you can often get away with a slight skew (~1 year), but you can rarely get away with skews of more than 2-3 years.
[1]: https://developer.apple.com/library/archive/qa/qa1118/_index...
[2]: https://github.com/golang/go/issues/16606
[3]: https://support.apple.com/guide/security/trust-caches-sec7d3...
Fundamentally, containers are about namespace/isolation of a bunch of OS interfaces, so file system functions, network functions, memory management, process functions, etc, can all pretend like they're the only game in town, but crucially without having to virtualize out the kernel.
Does XNU have such namespacing functionality across all its interfaces?
Furthermore, the existing container ecosystem assumes a Linux syscall interface. [1]. Does macOS provide that? I expect not.
The way Docker Desktop (and podman.io) implement "containers on macOS" is a bit of a cop-out: they actually run a Linux virtual machine (using Hypervisor.framework/hvf), and have that just provide the container environment.
Is that what this project is doing? But then, how could it run a macOS container?
[1] based on the foundation that Linux, unlike BSDs, has a stable syscall interface!