You can run Linux containers with their own systemd's (obviously, they have their own PID 1). And, of course, that doesn't make them VMs.

If you don't trust a component running inside a Jail or a Linux container, you shouldn't be running it there; the kernel attack surface is big. At that point you've adopted the same security model as a phone jailbreaker has (except you don't get to dedicate hardware to the anti-jailbreaking problem the way the phone vendors do).

Well what are you going to do when you have to run something you don't trust?

Can't run TLS termination on an isolated host, because then the traffic to the origin goes over the network, which you also don't trust.

Don't want to run OpenSSL integrated into the daemon, because OpenSSL is garbage.

Couldn't run anything else in the immediate aftermath of Heartbleed, because GnuTLS was worse, and LibreSSL and BoringSSL hadn't been released yet.

If you really don't trust it, you run it inside a virtual machine.

The truth is, you really do trust OpenSSL. You just trust it less than other things. That's fine! Layer controls on top of it. In Linux-land, this is the point where you'd start thinking about things like seccomp-bpf.

A truly untrusted workload is, like, a compute job you've accepted from a SAAS customer; it's arms-length multitenancy. You can't share kernels in that situation.

Worth pointing out that VM escape is a thing too: https://en.wikipedia.org/wiki/Virtual_machine_escape

It is. It's a much smaller attack surface.

I'm unconvinced it is. All of those virtual hardware drivers require their own code, often at kernel privilege. Hypervisor functionality at the processor level helps but is a huge attack surface in itself. X86 "rings" are under tested because it's hard to - many violations just kill your whole VM, but more importantly there's a ton of instructions that access "physical" memory and the state of the page mapper, even at the ring-0 level, matters a lot.

I do think VMs are likely more secure than containers in a cloud environment, because of course at that level you have both problems, but I don't believe that.the number of vulnerabilities found at the VM layer is at all reflective of their actual vulnerability.

It simply is. Even in a heavyweight VM with lots of hardware support, those hardware drivers are a tiny fraction of the user/kernel interface, no matter how you choose to count it (lines of code, number of foreign calls, number of exposed modules).

If you don't want to derive this axiomatically, fair enough: count vulnerabilities. The tally you're looking for is every Linux LPE versus every Linux KVM escape.

> I currently run two daemons in a vnet jail to get a separate network for those daemons, but chrooted to /

When you do this, do the jails/chroots act like separate overlays on top of / to the daemons?

For example, if you use systemd-nspawn or systemd-run, you can run daemons in containers that get their own view of /, but writes take place in separate overlay file systems.

I'm curious how that compares with what you're doing with jails.

No, my jailed deamons don't have an overlay, writes manifest to the filesystem the same as if they weren't jailed (which is good for me). Someone elsewhere described a vps they user which did have an overlay system, but I haven't used that.