Indeed, LXD would seem to be a faster, more efficient way of doing what you would do with traditional VMs. I don't see a lot of overlap with docker, despite the superficial similarity.

There are still a few gotchas on the low level stuff, especially networking, but also other areas that involve kernel drivers more directly.

The security around the containers still isn't as good as a hardware VM, but I think hardware segregation is on the road map.

It's ideal for running container inside VMs, though, and offers nice management segregation on those larger VMs.

Can you name some of these networking gotchas you mentioned? And elaborate on the security remark?

The major security issue is that containers share the same kernel as the host, with the whole kernel syscall interface as attack vector. VMs run separate kernels that only interface with the host via the hypervisor.

Well, there's also the issue that "Linux containers" are not a kernel primitive. So there's plenty of syscalls that aren't even namespaced (the keyring ones come to mind). Docker handles this by having a default seccomp profile that disables a bunch of syscalls that aren't namespaced.

However, the development of Zones and Jails (on SunOS/Solaris/illumos and FreeBSD) was much more security focused, with the default being "that's not allowed in a (Jail|Zone) until we can make sure it's safe". I really wish Linux had just ported Jails or inspired their security model on them.

I do a lot of work with the internals of Docker and runC at SUSE. Trust me, it's not pretty how you have to set up "Linux containers" and there's 1001 gotchas.

And we haven't seen many hypervisor security holes in the past... :) Especially with Xen.

> I always hated that docker forced you to treat a container as a single process and spoke about it as if it's a virtue. Let me decide how many or little processes I want running in a container.

You can have as many processes as you like running inside a Docker container, you just a need a single process to "boot" them, even if it's a simple shell script.

The big difference is that LXD can run, in an unprivileged fashion, a systemd based distro without issue, and with a sensible view of /proc.

Docker can't currently do that, largely because running a distro in a container isn't an important goal for them.

Docker can run systemd in unprivileged container: https://github.com/vlisivka/docker-centos7-systemd-unpriv

PS.

Try it:

    docker run -dt --stop-signal=$(kill -l RTMIN+3) -v /sys/fs/cgroup:/sys/fs/cgroup:ro --name centos7_systemd vlisivka/docker-centos7-systemd-unpriv

My impression is that someone created a specific dockerfile to get around the issues. I'm noting that LXD can run, for example, stock debian.

This appears to be a fairly complex subject. See https://news.ycombinator.com/item?id=11216950 and the original LWN article that it depends from.

There are quite a few issues with running a systemd-based distro in a container. The first being that /sys and /proc are basically not namespaced. So if you're not careful, systemd might start modifying your host. There's some work to improve this (the cgroup namespace -- yes, that's a thing now) but the point remains that allowing a root process to start touching /proc and /sys inside a container won't end well. I believe that LXD tries to get around this problem by mounting a FUSE filesystem on /proc and /sys that does a bunch of fakery. I don't like that, to be honest. It should be the kernel's job to namespace things properly.

That's not a good idea, for the same reason you don't want a shell script for /sbin/init.