rmolina

Appreciate your comment lmarcos!

Yes, the above instruction runs on macos too as long as you have docker-desktop installed.

You are probably referring to Sysbox (https://github.com/nestybox/sysbox), which I believe will meet your requirements (systemd, inner containers, security, etc).

Btw, Sysbox is already supported in Docker-Desktop (business tier only), so you can easily do what you want with this instruction:

``` $ docker run -it --rm -e SYSBOX_SYSCONT_MODE=TRUE ghcr.io/nestybox/ubuntu-focal-systemd-docker:latest bash ```

Disclaimer: I'm Sysbox's co-creator and currently working for Docker.

Thanks a lot Peter!

Right, that will work.

I fully agree that Sysbox use-cases extend beyond docker-in-docker and k8s-in-docker. These, docker and k8s, were just the first two system 'apps' that we decided to support, but Sysbox can grow to support many others.

To be fair, we are not expecting Sysbox to be able to run _every_ application, for many apps you'll still need hw virtualization. Our goal is to focus on the (potentially large) subset of apps that could technically run inside a container, but are not capable of doing so due to current runtime limitations (mainly coz they were not designed for that purpose).

Thanks.

We are actively working on this one as @ctalledo mentioned. Please ping us offline if want more details.

Thanks!

Please see our response to a similar question below. Hope that helps. Thanks.

"The main difference is that Sysbox is OCI-based, so it works with Docker/containerd and hopefully K8s soon (we are working on the latter). Also, correct me if I am wrong, but I don't believe LXD runs K8s inside without privileged containers. Having said this, I know LXD and Sysbox use many of the same OS-virtualization techniques to do what they do. And in fact we owe much of the work we've done to the ground-work done by the good folks at Canonical/LXD."

Thank you. Please reach out to us through email/slack. Would love to hear more.

Didn't know that. But it makes sense given that podman already supports it. Btw, i did a quick search but couldn't find anything on this (docker's systemd support in rhel). If you happen to know where to find these patches, please send them my way. Thanks.

Thanks @sanketdasgupta for reporting the issue and @asadlionpk for the explanation. Nestybox's github account is an organization now.

Thanks @geofft, you made a lot of great points.

I don't think rootless approach is fully aligned with what we're doing right now. True, we both rely on user-namespaces, and we both emphasize the security angle, but our goal is to expand the number of applications/functionality that can run in containers, which is something rootless approach may struggle with for some time.

Please see here if you haven't done it yet: https://github.com/nestybox/sysbox#sysbox-is-not-rootless-do...

In regards to our dependencies, we can operate with or without shiftfs. In both cases user-namespaces are always utilized. The rest of your approximation is correct: we need most of what you mentioned in your second paragraph, which btw, is already there (thanks to Canonical/LXD folks) starting in Ubuntu 5.0+ and 5.5+ for other distros. As you know, shiftfs is only present in Ubuntu at the moment, but as i said, we can live without it.

Which leads me to your question: why would you wait if the functionality you're after is already there? If having dockerd running as an unprivileged user is not a real must-have for you, then Sysbox provides a fairly secure solution while giving you all the functionality.

Sorry, i'm not familiarized with vpnkit yet, will take a look.

Correct, those are some of the benefits Sysbox-EE offers at the moment. That, plus efficiency & scalability features and hardened testing.

Thanks a lot for your detailed feedback @geofft. Please ping us on slack anytime.

Interesting. Thanks @jdoss!

We haven't had enough cycles to look at Podman in details (yet), but my understanding is that Podman and Docker serve similar purposes: they are high-level runtimes. (i'm obviating important nuances though and i'm not podman expert).

Sysbox, on the other hand, acts as a low-level runtime (same as runc), so we could potentially integrate with Podman too. In fact, we could _potentially_ integrate with anything that speaks OCI spec.

Having said that, we are not there yet, as for example sysbox wouldn't work with Podman in rootless mode right now; it should work in regular mode though, but we haven't tried it yet. If we accomplish this, it would allow podman to launch a larger set of applications too, same as we are doing for Docker.

As @wh33zle mentioned, CI/CD is an obvious use-case. Development environments is another one (pls see the other question i just answered on this topic).

But i also see 'production' scenarios. Think about having a reverse-proxy system container, inside of which you host the backend applications, which happen to be docker containers.

In essence, Sysbox allows you to create hierarchies of containers in a secured fashion, without the need to abuse 'sidecar' pattern, nor priv containers.

Absolutely, that's one of Sysbox's main use-cases, we usually refer to it as 'docker sandboxes'. As you mentioned, the idea is to have your entire dev environment within your fully-customized container, which would allow you to take that environment wherever you go, you are not tied to a particular hypervisor or cloud service-provider. We have heard of people already using Sysbox in Jupyter ecosystem.

I see. Will look into that right away. Thanks!

Thanks for the kind words!

Thanks!

We are out of the critical path, meaning we only emulate interactions with procfs / sysfs, and we only intercept mount syscalls at the moment, so we don't see any tangible performance hit. Having said that, we haven't done large scaling&perf tests yet.

Yes.

Having said that, we have certain limitations at the moment (e.g. we don't run all cni's), but we are not relying on priv containers as i believe is the case for existing K8s-in-docker solutions (pls correct me if i'm wrong in k3d's case).

Btw, we haven't tested k3d.io & sysbox interaction yet (part of our roadmap), but i believe we should be able to interoperate after making some minor adjustments. That would allow existing k3d users to operate in a more secure environment.

Sorry, not sure i got that. Can you please elaborate?

Got it, thanks for the explanation. I clearly see the use-case, just need to review cgroup specs (specifically cpuset) to fully understand if what you mention is already supported (which i believe it is).

Thanks! Both of your suggestions sound very interesting. I personally like the idea of creating large network topologies with a very few outer containers: the real mesh would be at L2/L3 levels. You would launch your large topo with just a 'docker run'.

The numa-aware idea would take some more research on our part, but at first glance looks like something that we could definitely explore.

Right, systemd uses pid 1, but it does so within the pid-namespace of the container, so each container has its own systemd. Hope it makes sense. Thanks!