Jails are a general mechanism to run multiple userlands on one kernel. They started with a change root directory, an IPv4 address and no access to anything dangerous (e.g. /dev/mem). Jails grew a lot features with the biggest bump in FreeBSD 8. Starting with FreeBSD 8 jails can be nested, have multiple IP addresses etc. but they still lacked System V IPC. This changed with FreeBSD 11.

Now the only thing jails can't have is their own IP stack. Jails share the hosts IP stack which improves efficiency and simplifies most deployments, but it prevents them having administrative access to the IP stack. There is an experimental kernel feature (VIMAGE) to jails to run their own instance of the IP stack but there are still some nasty bugs hiding in this code, because nobody thought about how to tear down the IP stack. After all it was initialised once during the boot process and kept running until the power went out.

The largest difference is in the mindset behind jails. Jails are designed as secure operating system level virtualisation. Docker on the other hand is fairly fragile and offers neither secure isolation between containers nor between containers and the host. Jails can contain a complete userland and this a very common setup.

A full FreeBSD userland + some ports/packages to make it useful is about one 1GB. This used to be a lot 15+ years ago when jails where created and the older jail management tools like ezjail reduce the per jail storage requirements with nullfs and unionfs hacks. These days 1GB isn't that much for a simple container and most FreeBSD servers run on ZFS. ZFS offers a much simpler and cleaner way to reduce storage requirements: just clone a snapshot (the template) create a new jail and copy a few config files into the clone. The only problem is that you can't rebase your clone.

Docker is designed around the idea of single purpose containers without stable storage. You can use jails to implement this idea, but FreeBSD jails support more than that. Also all the FreeBSD jail managers I used try to stay out of your network configuration as far as possible and at most configure alias IP addresses on existing interfaces.

Docker is very opinionated software fighting against limitations imposed on it by the Linux kernel. Jails are FreeBSD kernel feature touching multiple parts of the kernel with a minimal userland interface in the FreeBSD base system . Multiple higher level jail managers are available in the FreeBSD ports tree.

There is no reason why you couldn't implement a docker like jail manager and the jetpack projects started doing exactly this. Keep in mind that docket images are the worlds new statically linked binaries for people who can't figure out how to define and reproduce the relevant parts of their development environment in their production environment. Executing existing docker images with their Linux binaries would probably require a massive update to the Linux compatibility layer (a reimplementation the Linux syscall ABI).

Actually, Bjoern Zeeb (bz@) did a lot of work around VIMAGE (sometimes also called VNET) in order to track down these bugs and fix them for 11.0-RELEASE. If you haven't, you should give it a try, because it's now the way to use per-jail IP stack.

I did and during my tests i found no more panics, but I ran into a bug in the IPFW tables logic. The fix hasn't been backported to FreeBSD 11.0 as an errata. Also the epair pseudo-interface used for connecting IP stacks tops out at about 4Gb/s while the loopback interface can handle >40Gb/s on the same hardware, because it uses just one kernel thread (I don't remember the pps).

I hope we'll get VIMAGE into GENERIC for FreeBSD 11.1. He completely refactored the IP stack shutdown logic. Now the layer are shut down from top to bottom instead of the other way around. This avoids most of the nasty locking problems draining the higher levels with their pointers to the lower layer resources (e.g. routes, interfaces) first. The remaining bugs won't be found without a lot more exposure.

I did get panics but I haven't been able to reproduce them. The only thing I can think of is create a jail without an assigned IP, start it, assign one, shut it down and start it up again.

The bug I ran into was https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212649 which turned out to be an other symptom of https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212576.

> Docker on the other hand is fairly fragile and offers neither secure isolation between containers nor between containers and the host.

Please elaborate... Otherwise, great summary.

Docker moves fast and breaks things without stabilising them first this post explains the problem fairly well: https://thehftguy.wordpress.com/2016/11/01/docker-in-product....

There have been lots of bugs (some could be considered design failures) allowing processes to escape from a docker container. Fixing those problems hasn't received the attention from the docker community I would expect from a serious OS level virtualisation community. In some cases it boiled down to "yeah just don't do that" or "no problem just put docker inside a VM".

Perhaps you have the knowledge to inform me on a question to which I have never been able to find a plain answer. Do FreeBSD jails have anything like UID/GID virtualization and namespace isolation and resource quotas that LXC/LXD gives you on linux via cgroups?

Why is this NEVER brought up when FreeBSD and linux are compared?

UIDs and GIDs are just numbers to the kernel. Root inside a jail become any UID/GID and chmod/chown files just like root outside the jail, but even root inside a jail is locked down into the jail root directory.

You can prevent root inside a jail from modifying files with several mechanisms. The simplest is to not allow (re-)mounting from inside the jail which is the default and mount the relevant file systems read-only. BSD extended flags offer a finer granularity and by default root inside a jail isn't considered privileged by chflags(2). You can disable the security feature in which case the normal rules apply and access is controlled by the secure level (jails have their own secure level).

Processes running inside a jail have access to the full UID and GID namespace without any mappings.

PIDs exist in a global namespace and are allocated by the kernel. Processes inside a jail can't address PIDs unless they are inside the same jail or a child jail. Root inside the host can always address all PIDs. Non-root processes in the host (and jails) are subject to the security.bsd.see_other_uids and security.bsd.see_other_gids sysctls which can be disabled to protect the obvious ways to spy on other users. There is also support for various forms of mandatory access control. Have a look at the `mac_*` manual pages for more information on the available MAC disciplines.

Restricting the jails to just their processes in a shared namespace is a much cleaner alternative to loosing the PID as unique global process identifier. And mapping between different scopes is even worse than extending the identifier from an integer into a tuple.

FreeBSD also supports hierarchical resource limits. Jail ids are one possible subject type to limit. That way you can limit resource consumption per jail. See https://www.freebsd.org/cgi/man.cgi?rctl and https://www.freebsd.org/cgi/man.cgi?rctl.conf for more details on that.

Every LXC/LXD container can have its own PID 0 init process, and none of them even have visibility of the others. That's what I am not seeing in jails. Root on the host cannot even see any of the LXC/LXD PIDs.

Am I not seeing it because I am not looking hard enough, because FreeBSD guys don't really understand this whole issue, or because it's not there?

Re: resource limits: can the CPU load per jail be limited?

What issue? Root outside the jail is implicitly trusted by the jail so why would you hide resources (in this case jailed processes) from the "true" root? Jails can't see each others processes. The only case I can think of how one jail could affect an other is if they are connected by a unix domain socket in that case you could probably send a process descriptor from one jail to the other and the capability represented by the descriptor should cross the jail boundary. This would require the sending to sendmsg() the descriptor on purpose and I wouldn't consider it a security problem, because both jail have to cooperate to create this condition and be configured to share an existing connected unix domain socket a shared writable file system subtree (e.g. through nullfs what Linux calls a bind mount).

Yes you can limit resources per jail through hierarchical resource limits.

Does this answer your questions or did I missunderstand your question?

Across jails, pids are not visible. I can't comment on the ability of a user on the root of a linux system being able to see the pids in a LXC, but I bet there is a way. After all they are running on the same kernel. Is the non-visibility of the proceses just a userland trick on linux?

Resource limits for Jails can be done with rctl

It's important to understand, when people talk about cgroups, they do mean not only CPU/memory limits, but also network and disk I/O limits, that freebsd doesn't offer out of the box.

The firewall can match on the jail id for local sockets and apply traffic shaping based on these matches. Hierarchical resource limits on CPU/disk/memory/... can be configured per jail.

Of course jails have UID/GID virtualization and namespace isolation, that's what the purpose of jails is. Cgroups-like resource quotas - not so much, but there are ways to limit resources.

Actually for resource quotas/management there is rctl. So resources can be managed.

This is a fantastic summary, thank you!