I've been using rootless podman containers for several years, but I've stopped using them recently. I read this security note on the Arch wiki [1]:

> Warning: Rootless Podman relies on the unprivileged user namespace usage (CONFIG_USER_NS_UNPRIVILEGED) which has some serious security implications, see Security#Sandboxing [2] applications for details.

Accordingly, I've since set sysctl `kernel.unprivileged_userns_clone` to `0` on my system which disables rootless containers, but is now supposedly safer.

The dialog on this is a bit controversial, it seems rootless podman is pardoxically both more secure and less secure at the same time. Rootless podman doesn't have root access, but with user namespaces turned on, podman has access to kernel apis that have not been rigorously tested for non-root users. Someone explained this on Security SO a bit more in depth [3]:

> The reason for this is that much of the kernel that is only intended to be reachable by UID 0 is not audited particularly well, given that the code is typically considered to be trusted. That is, a bug that requires a UID of 0 is rarely considered a serious bug. Unfortunately, unprivileged user namespaces make it possible for unprivileged users to access this very same code and exploit security bugs.

[1] https://wiki.archlinux.org/title/Podman#Rootless_Podman [2] https://wiki.archlinux.org/title/Security#Sandboxing_applica... [3] https://security.stackexchange.com/a/209533

Note: Every mainstream OS since RHEL8 has defaulted to User Namespace being turned on. Debian, Fedora, Ubuntu, RHEL, Centos ... all have this defaulted on. While overall the statements are true. There has not been many recent vulnerabilities forcing distributions to reconsider this setting. Podman is taking advantage of something that is turned on on most of the Linux systems running in the world. So saying this is not well tested on novel is a big exageration. User Namespaces have been available for almost 10 years and enabled for rootless users for at least 5 years in main line distributions.

Shouldn't the best security come from using Rootless podman, but configuring SE Linux to prevent all other binaries from using unprivileged namespacing?

The concern isn't that podman itself might use that extra attack surface (because, you give podman so much more rights by making it setuid root), but that other untrusted binaries (like a virus) might use unpriviledged namespaces to exploit the kernel.

TBH, by the time you are worrying about privilege escalation from userspace threats on a typical single-user linux machine, you have already lost.

A smart threat will just modify your $PATH or bash aliases to replace su and sudo with wrappers that execute their own commands the next time you use su/sudo to do something legitimate.

I’m not too knowledgeable about SELinux, but Podman itself can create unsafe (privileged, etc.) containers, so I don’t think naively restricting user namespaces to Podman is going to help, exploits could just create the namespaces through Podman instead of doing it themselves.

BTW, if I remember correctly Podman isn’t setuid root except some very small helper binaries, almost all of the process of creating the container can be done without root privileges.

On a workstation, I don't see so much problems. Considering that most people uses an user account that just by typing sudo (or of course spawining a process that invokes sudo) you become root without a password (because we are all annoyed inserting it, but even if we require one, it's trivial to intercept.

If it's a system you use interactively you have other problems. For a server yes, unprivileged containers are a very bad idea, not doubt.

I have found that the processes I put in motion on my workstation tend to creep into production..

I'm not sure what the tradeoff is. You can give a user access to rootless containers or you can give them root access to run containers. User namespaces are at least attempting to improve on the Docker situation where access to the Docker daemon is effectively equivalent to root access, unless that has changed recently.

Does the exposed kernel surface actually extend to the processes running in the contained environment?

You could also give an unprivileged user access to run VMs, that has its own set of security tradeoffs, but VM technology is generally considered more isolated and mature.

Unprivileged user namespaces give an unprivileged user the opportunity to use syscalls like chroot, mount, etc. which they historically couldn't use. Container managers like Docker and Podman drop those "capabilities" after creating the container, so an application running inside a container isn't that unsafe (probably, at least everybody is doing it nowadays for production servers). I think a more problematic scenario is that someone who gets hold of an unprivileged user can then create an user namespace without dropping those capabilities and then use a local root exploit based on those syscalls you normally weren't able to use.

I think the problem might be in the interplay of how many allications/daemons already try to drop privileges by running as a user after doing the few things they need those privileges for (like binding to a low port), and if you don't even need those privileges, you don't even need to start it as root, and your container has well known security properties.

If you're running rootless and using namespaces to allow non-root users access to previously root-only kernel APIs, then a bunch of prior assumptions may no longer hold, and there's a new attack space available to target that has always existed, but was of no use previously to exploit.

This is the concern that kept Debian from enabling user namespace support in their kernels by default, until their recent Bullseye release. It's not Podman-specific; it also affects Docker's rootless mode.

Can't the issue be avoided by simply launching the application as a non-root user within the container namespace?

> Can't the issue be avoided by simply launching the application as a non-root user within the container namespace?

Yes, though the issue currently is that with the likes of (rootfull) docker containers, users end up setting docker to not need to call sudo every time. Meaning, if an attacker accesses your workstation (or a server), they will be able to run docker containers as root. This may as well be giving the attacker access to root as you can really easily give yourself full root access if you can launch rootfull containers. Having (rootfull) docker run access is pretty much the same as having root access.

This sysctl `kernel.unprivileged_userns_clone` doesn't even exist in upstream kernels: the feature can only be disabled at build time.

So then what do you use for container that's better from a security standpoint?

What do you use instead?

I mostly use K8s on VMs[0], and docker the same way. Podman is a just a hobby for me, but I think running Podman in VMs is still the right way to go because it offers you a way of coding distributed systems the same way you can with K8s (with the cool style of systemd and single process containers:))), even if its just your laptop, you could move it to production clusters more easily, because you were forced to think distributed from the start. KinD[1] is also a good option if you can't or don't want to run VMs (but you have to run docker somewhere.. :/). The point is to simulate multiple nodes even when working in development.

[0] https://news.ycombinator.com/item?id=28395329 [1] https://kind.sigs.k8s.io/

I get why the pithy "chroot with more steps" comment has been largely dismissed, but when all you're going for is the temporary installation of build dependencies without impacting the permanent system, chroot really can do enough. The other stuff with user namespaces to isolate process IDs, networks, and map UIDs to get root in the user namespace without actually being root, aren't strictly needed.

There is an Arch Linux devtools project providing some shell scripts to automate setting up clean build chroots, but they're of course focused on setting up an isolated Arch system, and still require running as root. The true poor man's way to do this without being root is to use fakeroot and fakechroot, which is what the Debian builds do. The examples from Debian would all run debootstrap to set up a minimal Debian in the chroot, but you can just run "fakeroot fakechroot <any command>" to use your own build tools and bootstrap your own build environment. That way you don't even require a container runtime and don't need to open up the attack surface of user namespaces.

Along the same likes as fakeroot/fakechroot, I've found proot to be super useful for writing startup scripts for commands which make awkward assumptions about the filesystem layout.

Oh damn, I love this, thank you!

EDIT: Looks like they don't provide either an Ubuntu/Debian executable or an Ubuntu root image, which is a bit unfortunate. Odd that someone would make a project and ignore the most common Linux distro...

IME podman is optimized for use in Redhat based environment. I've had a couple of odd problems when trying to use it on Debian/Ubuntu.

It's not surprising given it's a Redhat project, but it does put me off using it as I'm most comfortable with debian derived environments.

It's pretty easy to create one yourself (from an open MR https://github.com/containers/toolbox/pull/483/files#diff-21...), but it would have been nice to have out of the box.

I've used buildah to set up a C/C++ dev environment in a Ubuntu toolbox at work without any major issues.

This sounds exactly like Singularity http://singularity.hpcng.org/

So they reinvented Nix except badly?

(Eventually people will come around to the Nix model. Especially because it's, essentially, like what happens when you install apps on phones.)

Apart from that both can be used for sandboxing/isolation, I see no similarities between the two.

Sounds like a chroot with extra steps

That's a way to describe pretty much all container things in a superficial way.

It's not even superficial. A chroot is a limiting case of a container that only rebinds the filesystem; or, equally, a container is a chroot that also rebinds the user namespace, process namespace, networking, etc.

Can you escape from a mount namespace? You can escape from a chroot.

You can change into an existing mount namespace, which is what commands like "docker exec" or "machinectl shell" do to execute a command within the mount namespace of an existing container. The container can do the same to escape its mount namespace, but it needs privileges to execute the respective namespace-changing syscall, and it needs to know which namespace to change into.

I have an application in a Kubernetes-managed container which needs to do exactly this to access the host system (to open LUKS containers and mount their contents), so I have it run in privileged mode (i.e. with access to all syscalls) and give it the full host filesystem as a bind mount, so it can do:

  nsenter -m /host/proc/1/ns/mnt mount $DEV $PATH

(The actual nsenter invocation is slightly more complicated, but that's the basic gist.)

Side-note: Since there will inevitably be a comment along the lines of "but then what do you even need isolation for", I'm very explicitly using Kubernetes as a deployment mechanism only. Other pods within the same Kubernetes are strictly isolated, but this one deliberately runs with practically no isolation. But I still get the consistent environment that Kubernetes provides (rolling upgrades, log shipping, liveness probes, etc.).

I know this was made tongue in cheek, but that was my first thought as well. Seems it would be more valuable for one to explain how and why it's better than a chroot, instead of redescribing how a chroot works but adding the word container.

It really just goes to what containers are. You can think of them as at a simple level as tars of chroots you can pass around and that an ecosystem of tooling has built up around.

This allows you to easily deploy generic ones you download from the community, and since they usually ship the script that was used to build it, it's easy to extend them with your own changes by creating your own script that uses a generic public one (the base, not the script) as a starting point, allowing everyone to share work. (e.g. get an nginx container and do the small bits to customize it for you in a script, and you don't need to worry about the building or system setup of container)

In addition to the normal chroot stuff, it uses cgroups functionality on linux to constrain it even more such as limiting CPU and RAM or disk IO, and can lock down a bunch of functionality from even root inside, and you also usually have to specifically allow whatever internal ports are listening externally (and not necessarily on the same port).

So containers are what you get when you take the idea of a chroot and try to extend and build upon it towards the direction of a true virtual machine, but without going to the step of fully emulating hardware. The benefit is that they are super lightweight and can more easily share CPU and RAM. The downside is that you're exposing more of your base system kernel to it, so it's likely somewhat less secure than a true hypervisor (but very likely more secure than just running those processes on the base system).

So, now that we have a shared base of what a container is (sort of, I'm not sure enough on the specifics to assume I didn't make some missteps in that description), rootless containers extend that so that a regular system user can run a container which internally runs as root, but not the real parent system root, but it works for the container because the kernel uses namespaces to allow it. This is both cool because it allows for root to be used even less on a system and scary because as another comment in this thread noted, it exposes some kernel APIs to "root" users (run and controlled by system users) which might not normally be checked as much for security problems since you have to be root already to use them.

Also: Julia Evans has a great writeup along these lines:

https://jvns.ca/blog/2020/04/27/new-zine-how-containers-work...

This is actually a really helpful write up - that makes sense. Thank you.

chroot only affects a processes view of the filesystem it doesn’t affect:

- the process’s view of other processes (i.e. ps isn’t isolated)

- the process’s view of the kernel’s VFS, (i.e. mount isn’t isolated)

- the process’s view of the networking stack (i.e. ifconfig and iptables aren’t isolated, and socket numbers are global)

- the process’s view of users on the system (i.e. uids/gids are global)

And so with time, the hostname, IPC, cgroups…

And more are planned on being added like the kernel keyring, and /dev/log.

“containers” means a lot of different things to different people but the main things people want are.

- resource bundling: “take everything your app needs to run, and tar it up so that it all stays together. Don’t make me worry too much about missing libraries.

- sandboxing: don’t make me have to worry too much about what else is going on the system when I run my thing and reduce the surface area for attacks.

- delivery (the big value add): give me a bunch of tooling to create that tar, version it, and get it running on a bunch of different systems without have to worry too much about the underlying OS, hardware, network setup, or storage.

And bonus for things like k8s.

- service bundling: give me a bunch of tooling to get many of those tars running on different systems and don’t make me worry too much about service discovery, load balancing, software crash recovery, hardware crash recovery, deployments, task scheduling, secrets, config files and more.

Some pointers if anyone is curious about chroot vs other isolation techniques:

https://fly.io/blog/sandboxing-and-workload-isolation/

https://devops.stackexchange.com/questions/2826/difference-b...

For further reading, searching for ⟬docker vs chroot⟭ yields a bunch of interesting articles.

chroot just isolates the file system.

A "container" (whether Docker or LXC or Podman) is a bundle of isolations that includes the filesystem as well as processes and users, and constraints on memory, CPU, networking, and kernel calls.

AFAIK Docker, LXC, Podman, etc. are actually all the same underneath: they use 'runc', which chroots into a directory (along with extra restriction/config via cgroups, bind mounts, etc.).

The differences between Docker, LXC, Podman, etc. are the config, management, etc. of containers; e.g. Docker abstracts over the chroot directories, using a set of content-addressed tarballs (called "images"), and provides commands for starting/stopping/listing/etc. the containers that are running.

Dating myself a little, but LXC came along well before runc. It predates Docker, and was actually wrapped by the first versions of Docker. I learned about LXC, and used it, after doing some research on how to build my own PaaS at work (pre-Docker) and reading that Heroku was using LXC for their platform. It worked decently for me, but required more in-depth knowledge of all the knobs for disk mounts and networking than Docker typically requires.

You may be right that LXC has switched over to runc since then. I haven't kept up. I would be surprised though.

I was actually being serious, your comment is more helpful though.