But if you can get isolation, security AND reproducible environments using a VM, specially one that's nearly as fast as a OS process, the case for using containers instead pretty much disappears. I don't know this LiteVM thing but I will definitely investigate that, specially given that on my Mac I need to use a VM anyway to run containers!

Check out kraft.cloud and the accompanying LF OSS project www.unikraft.org :) (disclaimer: I'm one of the authors of the paper and one of the people behind that cloud offering). On KraftCloud we use Dockerfiles so users can conveniently specify the app/filesystem, and then at deploy time transparently convert that to a unikernel (specialized VMs). With this in place, NGINX cold starts in 20 millis, and even heavier apps/frameworks like Spring Boot in < 300 millis (and we have a number of tech to bring these numbers even further down).

Looks very cool.

For anyone else wondering how heavy this is on a MacOS, I ran the install script and it just delegated to brew... brew listed the following packages being installed:

    ==> Fetching dependencies for unikraft/cli/kraftkit: aarch64-elf-binutils, gmp, mpfr, aarch64-elf-gcc, coreutils, gettext, readline, gawk, gnu-sed, pcre2, grep, make, capstone, dtc, mpdecimal, ca-certificates, openssl@3, sqlite, [email protected], glib, libunistring, libidn2, p11-kit, libnghttp2, unbound, gnutls, jpeg-turbo, libslirp, libssh, libusb, ncurses, snappy, vde, qemu, socat, wget, x86_64-elf-binutils and x86_64-elf-gcc

Most should already exist on your mac if you do development... it seems to rely on qemu, unsurprisingly... openjdk as well (probably to support Java out-of-the-box?), imagegick etc.

Took a few minutes to finish installing... the CLI seems to be based on the Docker commands (build, clean, run, 'net create', inspect etc.), some package-manager like commands ('pkg info', 'pkg pull', 'pkg list' etc.), a bunch of "cloud" commands (I suppose that's the non-free part) and "compose" commands just like docker-compose. Interesting stuff.

Note for the parent commenter: the Lua link in the landing page is broken: https://github.com/unikraft/catalog/tree/main/examples/http-...

I tried to run the C hello world example... I get an error, it wants to run Docker?!?! I thought the whole point was to avoid Docker (and containers)??

Here's the log:

i creating ephemeral buildkit container W could not connect to BuildKit client '' is BuildKit running? W W By default, KraftKit will look for a native install which W is located at /run/buildkit/buildkit.sock. Alternatively, you W can run BuildKit in a container (recommended for macOS users) W which you can do by running: W W docker run --rm -d --name buildkit --privileged moby/buildkit:latest W export KRAFTKIT_BUILDKIT_HOST=docker-container://buildkit W W For more usage instructions visit: https://unikraft.org/buildkit W E creating buildkit container: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?: failed to create container

PS. running the hello-world pre-built "image" worked:

> kraft run unikraft.org/helloworld:latest

EDIT:

A lot of stuff looks broken on MacOS.

For example, `kraft menu` doesn't work (error "no choices provided", even though the docs show it working fine without "choices"?)...

`kraft run --elfloader loaders.unikraft.org/strace:latest ./my_binary` also doesn't work (the docs show it working). Error: "unknown flag: --elfloader".

Seems like the product is still in alpha?!

Is the entire kraft stack open source? Can I run this on my own hardware?

Also was my question.

VMs in general use more CPU power as you have two OSes each doing things like updating their real time clock... There are VM aware OSes that will not do this, but it needs special code and CPU support which means you are often lagging behind the latest (to be fair this is rarely important) A container will normally be slightly faster than a VM never slower (assuming a reasonable OS - I can write an exception if I was malicious) and so there is a lot of interest if they are good enough.

At the time of publication of the article the tool used to create the minimalistic VM Tinyx was not released and as far as I can see was never released.

Correct, we never did release Tinyx, mostly because it was in a very unclean/researchy state = not ready for public consumption. In retrospect, we probably should have either (a) made it available in whatever state it was in or (b) put more cycles into it.

But then also monitoring gets harder. With containers you could see (and monitor) all those processes running in container easy on the host.

With VMs? You now need a way to get data from within the VM which in most lightweight VM implementations just not possible.

On kraft.cloud we have unikernels (specialized VMs) with Prometheus exporters that can be scraped, and other monitoring facilities.

I wouldn't discount the massive user base of containers among developers.

i think at that point it's kind of just semantics

i don't think devs care if they use containers or VMs, as long as it's easy and they don't have to worry about which version of Python the host is running

This. It's why vagrant was popular before the container revolution.

The killer app of Docker isn't the container, it's the depth and uniformity of the UX surrounding the container system. When that is broken by something on the host (non x86 cpu was a major pain for a while before popular images were x-built) and emulation gets in the way and is not as easy, or just mildly different (windows behind corporate firewalls that assign ips used by the docker engine for example), the ease of use falls away for non-power users and it's all painful again.

Tech like Docker for windows and Rancher Desktop and lima has largely matured at this point, but somebody could make a new machine and then the process of gradual improvement starts all over again.

No it didn't. I want to rebuild my image using a different base.

Docker? One line possibly.

VM? Afternoon (unless I want to write ansible too)

In this sort of scenario, then you can also use something like cloud-init: https://cloudinit.readthedocs.io/en/latest/

Or I could use the tool I currently am. Your reply doesn't solve my use case it just says use yet another tool.

You don't need a VM on Mac to run containers, check out OrbStack, they provide a Docker compatible engine that is using native MacOS capabilities for running containers without the hidden Linux VM.

I don't know where you got that idea. OrbStack absolutely runs a Linux VM. That Linux VM then uses Linux containerization technologies (namely LXD) for each separate OrbStack 'machine' you set up, which is how you get such fast startup times for your OrbStack 'machines'.

For Docker, OrbStack does the same thing as Docker Desktop, Podman Desktop, Rancher Desktop, etc., which is set up a Linux VM running Docker and then present a native socket interface on macOS which relays everything it receives to the Docker socket inside the VM.

macOS doesn't have native capabilities for running containers, which is why the nearest thing you can get to containerd on it requires you to disable SIP so it can use a custom filesystem to emulate bind mounts/null mounts: https://darwin-containers.github.io/

If you read the PRs where the principal author of the Darwin Containers implementation is trying to upstream bits of his work, you'll see containerd comparing his approaches to others and complimenting them by calling them 'the most containerish' because real capabilities aren't there.

(I believe I've read rumors here on HN that Apple has those features internally, fwiw. But they've evidently never released them in a public copy of macOS.)

Another clue in all this is to just run uname in any of your Docker containers in OrbStack; you'll see they're Linux machines. Some operating systems have Linux syscall emulation layers (WSL1, FreeBSD's Linux emulation, Illumos' LX Zones) that could perhaps be used to run Linux containers without hardware emulation or paravirtualization in combination with some native containerization capabilities. Afaik Illumos' LX Zones is the only implementation where that's a supported, intended use case but maybe FreeBSD can do it. At any rate, macOS has never had that kind of syscall compatibility layer for Linux, either. So when you run `uname` in a 'macOS container' and see 'Linux', you can be certain that there's a VM in that stack.

PS: Aside from the fact that it's proprietary, I really do quite like OrbStack. It's the nicest-to-use implementation of something like this that I've tried, including WSL2 and Lima. The fact that it makes the VM machinery so invisible is very much to its credit from a UX perspective!

Interesting! I'd swear that in the early days of OrbStack somewhere on their website I've read they're using native MacOS frameworks without the need of Linux VM, but I can't find that anymore (they don't mention Linux VM either, but the language still differs from what I remember).

They do use native GUI frameworks rather than something like Electron, which they still mention. And maybe they also used to have something about relying on Apple's Virtualization Framework or something like that, rather than qemu as Lima used for a long time. (I think it may still be Lima's default, but not for long.)

Certainly depends a lot on what the term "VM" actually means in the context. If it's something as specialized as the JVM, or a native virtualization with an extremely trimmed down guest, then at some point you'll find yourself in need of something more heterogenous, e.g. running a tool on the side that does not fit the VM. Then you're back at square one, only this time with containers (or back in some irreproducible ad-hoc setup). Going with containers from the start, containers that may or may not contain a VM, and that may or may not actually do more than what a VM could supply, that's much less hassle than changing horses at a later point.

Where is the tooling to build and distribute lightweight vms like containers? How can I copy one html file into an nginx VM, built this vm image with multiple architectures (I have arm and x64 servers), publish it, pull it, and run it multiple times?

Once again: Containers are not about isolation or security, they are a package format for shipping applications. The packages are easy to build, distribute, multiarch, ...

And requiring a Linux-VM on macOS to run Linux containers, is not particularly surprising.

> specially given that on my Mac I need to use a VM anyway to run containers

Take it up with Apple.

I agree in principle, but this is not so easy when shared disk IO is introduced, in my experience.

[disclaimer: I'm one of the authors of the paper] I 100% agree, containers are an amazing dev env/reprodicble env tool! In fact, we think they're the perfect marriage to the unikernels (specialized VMs) we used in the paper; on kraft.cloud , a cloud platform we built, we use Dockerfiles to specify apps/filesystems, and transparently convert them to unikernels for deployment. The end result is the convenience of containers with the power of unikernels (eg, millisecond cold starts, scale to zero and autoscale, reduced TCB, etc).

Containers are perfect for build environments and for creating the root filesystem. The issue is that the kernel these days are super bulky and are intended for multi-user, multi-process environments. Running a container runtime on top just makes it worse when you're looking for "isolation".

This paper argues that when you build a extremely minimal kernel (i.e. ditch Linux entirely) and link your application against necessary bits of code to execute _as_ a VM, then you'll get better performance than a container and you'll get that isolation.

This is in fact true based on performance studies, the follow up paper to this shows so: https://arxiv.org/pdf/2104.12721

(Disclosure, co-author of the linked paper.)

We ended up taking this to real workloads if you want to see it in action: https://unikraft.io/

Your pricing page has a mistake in the Free tier. The number 1 which is supposed to be a superscript is instead shown as HTML markup (¹).

Can confirm it's present on mobile.

I am looking at the examples. They all have a Docker file. If that just for local development on my laptop?

Using the deploy command line tool is the Docker file used to determine dependencies for the hosted VM? What if a developer is using an unusual programming language, like Common Lisp. Is that doable?

A Dockerfile is just a file with a bunch of commands to execute and get a working "computer". https://github.com/combust-labs/firebuild is fairly aged translation of the Dockerfile to a VM rootfs.

> build a extremely minimal kernel (i.e. ditch Linux entirely) and link your application against necessary bits of code

It would be nice, but this is really hard to do when modern software has so many layers of crud. Good luck getting say, a PyTorch app, to work doing this without some serious time investment.

But you don't need to write against all the layers of crud. You only have to write against the bottom layer, the kernel API. This sort of software would have no need to specifically support "libxml" or "TLS", because that is multiple layers above what this sort of software does.

The flip side is that if you want something like low-level access to your specific graphics card you may need to implement a lot of additional support. But of course nothing says you have to use this everywhere at the exclusion of everything else. There's plenty of systems in the world that from the kernel point of view are basically "I need TCP" and a whole bunch of compute and nothing else terribly special.

[Author of the paper here] You hit the nail on the head, this is precisely what we do (kernel API compatibility) with the LF Unikraft project (the evolution of the 2017 paper) at www.unikraft.org, and kraft.cloud, a cloud platform that leverages Unikraft.

Most of that effort should be sharable. if you know you will only have one python process you can get rid of a lot of cruft. If you know you will be running in a VM then you only need the driver for the network interface the VM provides not every network interface every designed (often including ones that your hardware doesn't even physically support). So while there is serious time investment it isn't nearly as much as it would be to write a competitor to linux.

I'm not sure if I missed a bit here, but I have some colleagues doing research on unikernels for HPC and the point is that this unikernel is running directly on the hardware or hypervisor and not inside another VM. The unikernel is effectively a minimal VM and the network stack is one of the things they struggle the most with due to sheer effort.

[One of the authors of the paper] I wouldn't recommend writing a network stack from scratch, that is a lot of effort. Instead, with the Unikraft LF project (www.unikraft.org) we took the lwip network stack and turned it into a Unikraft lib/module. At KraftCloud we also have a port of the FreeBSD stack.

> Running a container runtime on top just makes it worse when you're looking for "isolation".

The point of the poster was pretty clear:

“The main benefit is not isolation and security”

If this were true, then wouldn't folks just need an application binary that statically links all of its required libraries and resources into a giant, say, ELF? Why even bother with a container?

First thing that comes to mind is the need to link against libraries across platforms. Imagine that my app depends on opencv, if I wanted to statically link everything on my Windows machine, I need to compile opencv for Linux on my windows machine (or use pre-compiled binaries). Also, if you link against libraries dynamicaly, it's likely you can compile them on the host machine (or in a container) with more optimizations enabled. And the last thing is probably the ability to "freeze" the whole "system" environment (like folders, permissions, versions of system libraries).

Personally, I use containers to quickly spin-up different database servers for development or as an easy way of deployment to a cloud service...

Programmers discover the benefits of static linking, and then programmers discover the benefits of dynamic linking, and then programmers discover the benefits of static linking, and then...

Anyway containers go quite a bit further than just static linking, most people aren't out there linking all the binaries that their shell script uses together?

Well yes, but try turning some random python, java or ruby service into a single binary .. now do that 12 times. Or try with a native app that leverages both the GPU and libLLVM, and enjoy finding out the kind of precautions you have to take for LLVM to not blow up on a computer where your GPU driver was built with a different LLVM version.

That's exactly the summary of this.

That said, it makes sense from a developer POV; if, during development, you don't need the isolation you can run multiple containers (with on paper fast boot times and minimal overhead) on your development box.

There's plenty of cases to imagine where you need the containerization but not necessarily the isolation.

That what I do. I use https://github.com/jrz/container-shell so my projects are contained, and I'm a little bit protected against supply chain attacks

Why not use bubblelwrap or selinux or something? Docker is overkill for this.

Haven’t heard about bubblewrap, and I’m on my macOS. But I’ll look into it for inspiration

Because static libraries ain't a big thing anymore. Maybe they will become popular again. This would make it easier to have reproduceable build without a container. But I think containers are the new static libs now

Interpreted languages and their associated dependencies are more of an issue than static linking with compiled languages.

glibc doesn't work statically linked and lots of stuff depends on glibc

Yes, indeed, and that is golang's main reason for existing.

people would absolutely love to do that, but it's difficult, and the UX sucks

so people use containers instead (even if the container literally only contains a single statically linked binary)

What if you application is not just one binary. What if it's a pipeline of complex tasks, calls some python scripts, uses a patched version of some obscure library, ...

It's not possible to package half a Linux distribution into a single binary. That's why we have containers.

> If there is some additional isolation required, just run the container in a VM.

No. Running a container in a VM gets you no additional isolation. Containers share kernel space and as such have limited isolation to VMs, which have isolated kennels. In exchange for this Lack of additional isolation, you’ve added a Bunch of extra Complexity.

Pardon the extra caps I am using iOS voice dictation.

I think they mean run a VM with one container inside. So you do get strong isolation.

This is similar to how managed container IaaS works. They launch a VM and run your container in it.

It is extra complexity but has a few advantages. 1. People already have a convenient workflow for building container images. 2. The base OS can manage hardware, networking and whatever other low-level needs so that the container doesn't need to have these configurations. 3. If you want to trade of isolation for efficiency you can do this. For example running two instances of a container in the same VM. The container doesn't need any changes to support this setup.

The model of a single container within a VM just adds overhead. The ideal case would be to remove the container layer and have the application(s) within the container run directly in the VM (which hopefully only includes the libs and OS modules needed for the app to run, and nothing more).

This is the approach we take at kraft.cloud (based on the LF Unikraft project): use Dockerfiles to specify app/filesystem, and at deploy automatically convert to a lightweight VM (unikernel) without the container runtime/layer.

> So you do get strong isolation.

No, you don’t. There is no benefit the container is providing, because The only feature of the container is isolating you from the zero other containers running on the VM.

The isolation I am referencing is from the VM, not the container. Containers don't provide strong isolation, that is why the VM is required in this model.

The VM is an isolated environment itself. You do not need to be isolated from it.

Using two levels of userland isolation makes about the same sense as using 457 levels of userland isolation.

While reproducible build envs are a nice feature of using containers, they aren't the primary benefit.

The primary benefit is resource usage and orchestration.

Rather than duplicating entire aspects of an OS stack (which might ne considered wasteful) they allow for workloads to share aspects of the system they run on while maintaining a kind of logical isolation.

This allows for more densely packed workloads and more effective use of resources. This is a reason why the tech was developed and pushed by google and adopted by hyperscalers.

Containers have their place, if you’re racking and running your own gear life gets a lot better than screwing around with IPMI.

But reproducible infrastructure as code is just orthogonal to that: everything from Salt to Nix is credible in that role.

Containerizing on top of a Xen hypervisor never made sense to me.

All the younger engineers I talk to think you would need to be Albert Einstein to bootstrap a bare metal server.

As someone who made a living doing this at scale, where we would build a new datacenter every 2-4 weeks using 100% open source or off the shelf tools, I completely disagree.

I think PXE booting some servers and running a binary on them is 90% easier than most container orchestration engines, Kubernetes control plane, and all the other problems engineers seem to have invented for themselves. I also think it’s almost always much more performant. Engineers don’t have an intuition to realize that their XXLarge-SuperDuper instance is actually a 5 year old Xeon they’re sharing with 4 other customers. Cloud Prociders obfuscate this as much as possible, and charge a King’s ransom if you want modern, dedicated hardware.

I’ll urge that we don’t compare Albert Einstein to a k9s expert.

No one will ever be considered for a Nobel over Kubernetes.

There is no such thing as a reproducible build environment anymore. You can get a temporary reproducible build environment, but any sane security policy will have certificates that expire and that in turn means that in a couple years your build environment won't be reproducible anymore.

> but any sane security policy will have certificates that expire and that in turn means that in a couple years your build environment won't be reproducible anymore.

"Reproducible" is usually defined as "identical output except for the cryptographic signature at the end" (and that should be the only use for a certificate in your build environment, a high-quality build environment should be self-contained and have no network access). That is, once you remove the signature, the built artifacts should be bit-by-bit identical.

I said environment not build.

If you run multiple instances of a container image, you get a reproducible environment.

If you run a docker build multiple times, and copy a few files into the container, you get a reproducible container image. It is not a hash perfect duplicate, but functionally equivalent.

If builds of your favourite programming language are reproducible or not, is not really related to VM vs. Container.

NixOS and Guix System offer a far lighter and more reproducible approach, who also not push many running images build by unknown on the internet direct in production, full of outdated deps, wasting in the meantime storage and cpu resources...

Yet it doesn't even come close to a fraction of the adoption scale of containers, no matter how good it is. Ecosystems matter more than individual quality.

I think that's a gross overstatement. Ecosystems matter, yes. But they don't matter more than quality.

I think Betamax might have something to say about that.

That's because some interested parties have advertised containers, because they are good to sell as pre-built stuff, nice to sell VPS and alike etc, while pure IaC is useful for anyone and invite NOT to be dependent on third party platforms.

It's not a technical matter, it's a human, economical matter and actually... Most people are poor, following the largest scale means following poverty not a good thing.

You are absolutely correct, and the creators of Docker did mention that was the core reason. Unfortunately your comment comes 10 years too late for many.

we arguably already had this with things like python venv.

the articles main point still remains, containers are a slow and bloated answer to this problem.

I concede youll need containers for Kubernetes, and Kubernetes on the surface is a very good idea, but this level of infrastructure automation exists already in things like foreman and openstack. designs like shift-on-stack trade simplicity of traditional hardware for ever byzantine levels of brittle versioned complexity...so ultimately instead of fixing the problem we invoke the god of immutability, destroy and rebuild, and hope the problem fixes itself somehow...its really quite comical.

baremetal rust/python/go with good architecture and CI will absolutely crush container workloads in a fraction of disk, CPU, RAM, and personal frustration.

Python venv is language specific, doesn't handle the interpreter version and doesn't handle C libraries.

I really don't understand why people do this: I get having a distaste for containers but some people, seeing the massive success of OCI images, mainly seem content on trying to figure out how to discredit its popularity, rather than trying to understand why it's popular. The former may be good for contrarian Internet forums, but the latter is more practically useful and interesting.

I say this with some level of understanding as I also have a distaste for containers and Docker is not my preferred way to do "hermetic" or "reproducible" (I am a huge Nix proponent.) I want to get past the "actually it was clearly useless from the start" because it wasn't...

Not really, you'd still need a proper chroot / etc.

Check out https://github.com/jrz/container-shell

I tell people "An OCI container is a way to turn any random runtime into a statically linked binary."

It is very useful for managing dependency hell, or at least moving it into "API dependencies" not "Library dependencies", it is handy for pickling a CI/CD release engineering infrastructure.

It's not a security boundary.

(I'm 100% agreeing with parent, in case I sound contentious)

> It's not a security boundary.

It is a security boundary, just not necessarily the best one.

I would even claim it's a pretty good security boundary, good enough for most applications.

It's an incidental security boundary because CGroups happen to isolate the process fairly well.

Yeah, but that's not an incidental property of *namespaces* (of which cgroups is only one isolation axis), that was the requirement when namespaces were designed.

Yeah, I know. Namespaces are pretty cool outside containers too.

My comment was more of a soft jab against using containers as the ultimate "thing" for anything and everything. I prefer to use them as "statically linked binaries" for short lived processes (like document building, etc.).

But, whenever someone abuses containers (like adding an HTTPs fronting container in front of anything which can handle HTTPS on its own) I'm displeased.

Relevant XKCD: https://xkcd.com/1988/

All security boundaries are "incidental" in that sense, though. Virtualization isn't a "purpose-designed" security boundary either, most of the time it's deployed for non-security reasons and the original motivation was software compatibility management.

The snobbery deployed in this "containers vs. VMs" argument really gets out of hand sometimes. Especially since it's almost never deployed symmetrically. Would you make the same argument against using a BSD jail? Do you refuse to run your services in a separate UID because it's not as secure as a container (or jail, or VM)? Of course not. Pick the tools that match the problem, don't be a zealot.

> All security boundaries are "incidental" in that sense, though

X86 protected mode, processor rings, user isolation in the multi user operating systems, secure execution environments in X86 and ARM ISAs, kernel and userspace isolation, etc. are purpose built security boundaries.

Virtualization is actually built to allow better utilization of servers, which is built as a "nested protected mode", but had great overhead in the beginning, which has been reduced over generations. Containers are just BSD jails, ported to Linux. This doesn't make containers bad, however. They're a cool tech, but held very wrong in some cases because of laziness.

The motivation for MMU hardware was reliability and not "security". Basically no one was thinking about computer crime in the 1970's. They were trying to keep timesharing systems running without constant operator intervention.

Depending on your container runtime (Kata container or gVisor), it's a exceptionally strong security boundary too.

I agree with you, but the world seems to think otherwise!

Here are some projects that run docker containers on top of micro/lightweight VMs:

- https://github.com/firecracker-microvm/firecracker-container...

- https://github.com/kuasar-io/kuasar

- https://github.com/kata-containers/kata-containers

- https://github.com/QuarkContainer/Quark

- https://github.com/google/gvisor

- https://github.com/containers/libkrun

The main advantage in my use case is in fact isolation (network and volumes) and a well defined API enabling management of those containers in production (not k8s, a tiny subnet of that perhaps).

The isolation could be achieved using namespaces directly. But the API, tooling and registry add a lot of value that would otherwise require a lot of development.

Also last time I looked hypervisors aren't possible on all cloud vendors, unless you have a bare metal server. This matters in my case. Maybe it has changed in the past 3 years.

When docker fits it's great. Same can be said of k8s, where there are a whole bunch of additional benefits.

Swings and roundabouts.