Yes, you can run a vnet jail chrooted to /; same filesystem as the host, but a separate network system (you have to setup the network for the jail at this point; but I imagine the jail tools help with that). This is a fine use for jails. I currently run a vnet jail chrooted to / to do some crazy network stuff, but I just need a separate network, no other separation.
> * Can jails be used to limit memory/cpu/IO/network for subprocesses? For threads within a process?
I haven't used it, but it looks like yes/maybe with the rctl subsystem. This allows limits on lots of things, include memory of several types, cpu of a few types, filesystem io in bytes per second and operations per second, number of threads. Valid subjects for resource limits are process, user, loginclass and jail. Based on the manual, this doesn't let you limit network as you wanted, or place limits on threads within a process like you wanted. If you run a vnet jail, you can potentially set networking limits in other ways, but only if you pass virtual interfaces (such as epair, or taptun) to the jail rather than physical interfaces. This is a reasonable use for jails, but you might not need a jail for this?
> * Can live processes be moved into or out of a jail?
A process can move itself into a jail, but only if it's superuser. This is generally used administratively. Jailed processes can not be moved out of the jail, afaik; when the jail is destroyed, all processes within are killed. This is not within the design scope of jails.
> * Can jails be used to make a process think it's running as a different user?
I don't think so, but I'd use something with LD_PRELOAD to override getuid/geteuid for this; but I think I must not understand the question. You can certainly have entirely different users inside jail vs on the host?
Jails are basically like a VM-like/light: it appears like jail is its own system. Not quite as heavy as an actual VM under (e.g.) Qemu with virtualized hardware. A jail can have its own network stack:
* https://klarasystems.com/articles/virtualize-your-network-on...
You can then create a 'virtual patch cable' between the host and the jail and send one side of the 'cable' to the jail and do routing and stuff on the host:
* https://man.freebsd.org/cgi/man.cgi?epair
But a (sub)process cannot be 'sent' to a jail: jails 'boot up' like a normal system does, and so you'd have your PID 1 run your regular daemon startup.
> Can jails be used to limit memory/cpu/IO/network for subprocesses? For threads within a process?
Yes:
* https://wiki.freebsd.org/JailResourceLimits
> Can live processes be moved into or out of a jail?
No: jails are VM-like in functionality.
> Can jails be used to make a process think it's running as a different user?
The jail, being VM-like, would have its own passwd.
It's an option to run jails with a whole init and everything; then it's like a separate host. But you can also just run stuff in the jail, without a whole everything. I currently run two daemons in a vnet jail to get a separate network for those daemons, but chrooted to /, because they don't need a separate filesystem. At my last job, we ran a TLS termination proxy chrooted to a very limited directory, because we didn't trust OpenSSL after Heartbleed; the chroot had just the executable, ld-elf and the libraries it loaded, unix sockets to communicate with the origin server, and logfiles, most of which was chflags schg.
If you don't trust a component running inside a Jail or a Linux container, you shouldn't be running it there; the kernel attack surface is big. At that point you've adopted the same security model as a phone jailbreaker has (except you don't get to dedicate hardware to the anti-jailbreaking problem the way the phone vendors do).
Can't run TLS termination on an isolated host, because then the traffic to the origin goes over the network, which you also don't trust.
Don't want to run OpenSSL integrated into the daemon, because OpenSSL is garbage.
Couldn't run anything else in the immediate aftermath of Heartbleed, because GnuTLS was worse, and LibreSSL and BoringSSL hadn't been released yet.
The truth is, you really do trust OpenSSL. You just trust it less than other things. That's fine! Layer controls on top of it. In Linux-land, this is the point where you'd start thinking about things like seccomp-bpf.
A truly untrusted workload is, like, a compute job you've accepted from a SAAS customer; it's arms-length multitenancy. You can't share kernels in that situation.
I do think VMs are likely more secure than containers in a cloud environment, because of course at that level you have both problems, but I don't believe that.the number of vulnerabilities found at the VM layer is at all reflective of their actual vulnerability.
If you don't want to derive this axiomatically, fair enough: count vulnerabilities. The tally you're looking for is every Linux LPE versus every Linux KVM escape.
When you do this, do the jails/chroots act like separate overlays on top of / to the daemons?
For example, if you use systemd-nspawn or systemd-run, you can run daemons in containers that get their own view of /, but writes take place in separate overlay file systems.
I'm curious how that compares with what you're doing with jails.
The low-level details and fiddly knobs matter, not how it looks like from userspace when all is done.
Based on the jail(8) manpage I get the impression that jails are more like all the namespace and cgroup things rolled into a single entity. Which I guess makes it more difficult to use them incorrectly. But it also prevents other uses such as the one that jmillikin has in mind.
- Often boots user land like a VM (i.e. PID 1 is not the process you want to run, but whatever metal or VM freebsd runs as PID 1)
- Normally has its own network stack (VIMAGE)
- Normally runs a bunch of background services like a regular VM or on-metal would.
- Normally has the entire copy of user-land
- I've never seen short-lived jails either - you make a base dataset, clone it, but after that, you just run `freebsd-update` like you would in VM.
The majority of jail users treat them like lightweight VMs, only difference between VM and Jail from consumer standpoint: shared kernel, access to a subset of host's FS (sharing host FS to VMs in freebsd is not as easy as it is on linux).
To be clear, it's possible to use Jails like Docker, there is just no good tooling to do it. People would yell that whatever we have for jails is all we need with foam from their mouths every time you mention it.
The question that was asked what the technology can do, not how it's being used. What are the primitives, the atoms its made of.
People tend to use words that describe how they use things to describe what they are.
The question is not what you use a car for, but rather how the car is built.
The question was "Calling "VM-like" is not helpful because containers have also been called that, it also doesn't explain anything."
I've provided examples why "VM-like" is used to describe jails.
Technically, you don't have to. It's just because jails don't have convenient tooling around them (like docker or podman) it's easier to just boot it up like a normal system.
It feels like jails are still stuck with being described using older terms even after Linux containers and the distinction between them and VMs were mainstreamed.
of a VM.
The difference between "VM" and "VM-like" is the trailing modifier "-like". This means something is similar, but not the same thing. If it looks like a duck, but is made of rubber, we call it a "rubber ducky." If it looks like a duck, is wearing a blue shirt, has a speech impediment, and is not wearing any pants, we call it Donald Duck.
Thus the fact that jails, and docker look like VMs because they have their own PID 1, their own file system, their own slice of memory/cpu/IO/network; that is why people are, accurately, using the descriptor "VM-like" to describe them. People understand that docker is not a true virtual machine because it's not running its own kernel. In cases where it matters, pedantry between true VM vs fake VM, or VM-like (for eg security isolation between VM and docker container) is crucial, but most discussions where "VM-like" is used to help people understand OpenVZ, LXC, jails, docker, cgroups, etc aren't focused on the possibility of an RCE in the container escaping the container, but helping people understand what a container even is in the first place.
Or to put it another way, strcmp("VM", "VM-like") != 0.
>No: jails are VM-like in functionality.
Could this be implemented though if moving between sufficiently similar operating systems?
Seems like might be useful to imprison a suspicious acting process or to release once seen to be safe.
With all due respect, they are not.
The definition of a «virtual machine» is a settled matter, and the «M» in «VM» is important and is the differentiator: it allows one to run a different operating system kernel on the same host under the auspices of a hardware or a software supervisor (somewhat less of a defining feature).
Neither jails nor cgroups possess such a property, and both restrict users to the same host operating system kernel and its version, so none of them are «VM-like/light» irregardless of the semantic interpretation of the «-like» suffix.
VM-like functionality is provided by a different OS subsystem in both, Linux and FreeBSD, kernels.
For several of those questions about jails, the answer would be the same if you were just evaluating the questions against namespaces by itself.
FreeBSD Jails provide both secure isolation from the host and also provide the separated network namespace and resources with rctl(8) when needed.
To have Linux containers separated and secured you need additional layer for that - like SELinux or AppArmor. Only then Docker/Podman/other/... Linux containers are isolated and secure.
Regards.
To provide a security boundary between Linux processes, it's currently considered best practice to use something like Firecracker or gVisor.
That might be possible on DragonflyBSD. They have support for freezing a running process to disk and restoring the process again (potentially on another machine) [1], and they have support for jails, so I don't see why you wouldn't be able to freeze a running process on the host and restoring it in the jail or vice versa.
This may be me shouting into the void, but I wish there were an article directly comparing jails with namespaces, which is the Linux functionality that Docker uses. I can totally believe that FreeBSD jails provide a better / more unified / more secure experience than Docker, but to extend that into saying "FreeBSD jails are better than Linux namespaces" feels like a category error.
Questions I would like to see answered in that article:
* Can jails be used to run subprocesses in the normal filesystem, but with a different network environment (for example making a given command run its net traffic through TAP)?
* Can jails be used to limit memory/cpu/IO/network for subprocesses? For threads within a process?
* Can live processes be moved into or out of a jail?
* Can jails be used to make a process think it's running as a different user?
I feel like the answer to these questions is generally "no, that's not what jails are for", which is (1) a fine answer given the apparent goal of being a better chroot(), and (2) reinforces that jails and namespaces are addressing different problem domains.