ctalledo

Thanks again for the detailed response.

> Have a look at what gVisor actually does

I am aware of what it does, though I had missed the fact that the Sentry and/or Gopher run within a user-ns (could not find this in the docs). Had also missed the fact that it does perform procfs/sysfs emulation (makes sense), so I stand corrected on that. In light of this, I'll modify the Sysbox GH table to show gVisor as having a stronger isolation rating (in fact, our Sysbox blog comparing technologies [1] did give gVisor a stronger isolation rating).

> the sysbox approach is one kernel bug away from host system compromise

All approaches are one bug away from host system compromise (gVisor, VMs, etc.), though I agree that approaches like gVisor and VMs have a reduced attack surface.

> I've read numerous claims that sysbox is suitable for untrusted workloads

It's not a black or white determination in my view. Users choose based on their environments & needs. We always make it clear to our users that VM-based approaches provide stronger isolation, per the Sysbox GH repo:

"Isolation wise, it's fair to say that Sysbox containers provide stronger isolation than regular Docker containers (by virtue of using the Linux user-namespace and light-weight OS shim), but weaker isolation than VMs (by sharing the Linux kernel among containers)."

> Firecracker runs a full Linux kernel inside the VM, so it could always run regular Docker, Kubernetes or anything else

That's good to know (thanks), though the table in the Sysbox GH repo meant to compare Sysbox against Kata + Firecracker (since Kata is a container runtime). To the best of my knowledge running Docker, K8s, k3s, etc. inside a Kata container is not easy (see [1] and [2]).

> For containers, this used to be the case, but the situation improved in recent kernel releases.

It's correct that rootless docker/podman approaches are improving as far as what workloads they can run inside containers, although they still have several limitations [3], [4].

With Sysbox, most of these limitations don't apply because the solution works at the more basic "runc" level, Sysbox itself is rootful, and it uses some of the techniques I mentioned before (user-ns, procfs & sysfs virtualization, syscall trapping, UID-shifting, etc.) to make the container resemble a "real host" while providing good isolation.

Good discussion, please let me know of any more feedback.

[1] https://github.com/kata-containers/kata-containers/issues/20... [2] https://github.com/daniel-noland/docker-in-kata [3] https://docs.docker.com/engine/security/rootless/#known-limi... [4] https://github.com/containers/podman/blob/main/rootless.md

LXD is great, but one nice feature of Sysbox is that it's an OCI-based runtime, and therefore integrates with Docker, K8s, etc. In a way, Sysbox turns Docker containers or Kubernetes pods into LXD-like containers, although there are differences.

Thanks for the feedback; I am one of the developers of Sysbox. Some answers to the above comments:

- Regarding the container isolation, Sysbox uses a combination of Linux user-namespace + partial procfs & sysfs emulation + intercepting some sensitive syscalls in the container (using seccomp-bpf). It's fair to say that gVisor performs better isolation on syscalls, but it's also fair to say that by adding Linux user-ns and procfs & sysfs emulation, Sysbox isolates the container in ways that gVisor does not. This is why we felt it was fair to put Sysbox at a similar isolation rating as gVisor, although if you view it from purely a syscall isolation perspective it's fair to say that gVisor offers better isolation. Also, note that Sysbox is not meant to isolate workloads in multi-tenant environments (for that we think VM-based approaches are better). But in single-tenant environments, Sysbox does void the need for privileged containers in many scenarios because it allows well isolated containers/pods to run system workloads such as Docker and even K8s (which is why it's often used in CI infra).

- Regarding the speed rating, we gave Firecracker a higher speed rating than KubeVirt because while they both use hardware virtualization, the latter run microVMs that are highly optimized and have much less overhead that full VMs that typically run on KubeVirt. While QEMU may be faster than Firecracker in some metrics in a one-instance comparison, when you start running dozens of instances per host, the overhead of the full VM (particularly memory overhead) hurts its performance (which is the reason Firecracker was designed).

- Regarding gVisor performance, we didn't do a full performance comparison vs. KubeVirt, so we may stand corrected if gVisor is in fact slower than KubeVirt when running multiple instances on the same host (would appreciate any more info you may have on such a comparison, we could not find one).

- Regarding the claim that standard containers cannot run a full OS, what the table in the GH repo is indicating is that Sysbox allows you to create unprivileged containers (or pods) that can run system software such as Docker, Kubernetes, k3s, etc. with good isolation and seamlessly (no privileged container, no changes in the software inside the container, and no tricky container entrypoints). To the best of our knowledge, it's not possible to run say Kubernetes inside a regular container unless it's a privileged container with a custom entrypoint. Or inside a Firecracker VM. If you know otherwise, please let us know.

- Regarding "The claim that their solution offers large security improvements over any other solution with user namespaces isn't true". Where do you see that claim? The table explicitly states that there are solutions that provide stronger isolation.

- Regarding "The isolation offered by user namespaces is still very weak and not comparable to gVisor or Firecracker". User namespaces by itself mitigates several recent CVEs for containers, so it's a valuable feature. It may not offer VM-level isolation, but that's not what we are claiming. Furthermore, Sysbox uses the user-ns as a baseline, but adds syscall interception and procfs & sysfs emulation to further harden the isolation.

- "False marketing is a big red flag, especially for something as critical as a container runtime." That's not what we are doing.

- Rootless Docker/Podman are great, but they work at a different level than Sysbox. Sysbox is an enhanced "runc", and while Sysbox itself runs as true root on the host (i.e., Sysbox is not rootless), the containers or pods it creates are well isolated and void the need for privileged containers in many scenarios. This is why several companies use it in production too.

Hi HN, this is Cesar, one of the developers behind Sysbox, a next-generation "runc".

Sysbox enables containers (or pods) to act as "VM-like" environments, capable of running systemd, Docker, Kubernetes and more, seamlessly & securely.

Solves the problem of needing insecure privileged containers and complex container configs to run these workloads in containers.

It's a "runc", so it works under Docker and Kubernetes (and you can easily install it on GKE, EKS, AKS, Rancher, local cluster, etc.)

Very useful when using Docker-in-Docker or K8s-in-Docker (kind) for CI, when using containers as dev environments, or when running workloads that normally don't run in containers.

Hope you find it useful, would love to hear feedback!

Thanks! Yes, if you wish to setup dev environments backed by Docker or K8s containers/pods, Sysbox is an excellent way to do so because it gives you a rootless container inside of which you can run most workloads that run in VMs.

Prior to Sysbox this required privileged containers, which offer very weak isolation from the host (not to mention it also required complex container setups/entrypoints, all of which go away with Sysbox).

If you are using Docker-in-Docker, you may want to checkout the new Sysbox runtime (find it on Github). It's a new type of runc that sits below Docker and creates rootless containers capable of running Docker, systemd, K8s, etc. All you have to do is "docker run --runtime=sysbox-runc" <some-image-with-docker> and you'll get a docker daemon that is fully isolated from the host. It's a great way of avoiding privileged containers or mounts to the host docker socket.

+1 for the Emacs client.

Thanks; one thing I may have omitted mentioning is that Sysbox works with the fast overlayfs storage driver, meaning that when you do use it for Docker-in-Docker for example, both the outer Docker and the inner Docker are using overlayfs (as opposed to the slower vfs driver).

Agreed; it's certainly something we will keep in mind as we mature Sysbox.

A use case that we often get asked about for Docker-in-Docker is using the outer container as a dev environment that includes a developer's tools, ssh, and a dedicated Docker (CLI + daemon). It gives sys-admins a lighter-weight alternative to VMs for launching those dev environments, and works well in scenarios where efficiency & cost reduction is important and having VM-level isolation is not required. The problem is that prior to Sysbox, those outer containers had to be privileged containers, which provide very weak isolation (e.g., it's possible to turn off the host from within the privileged container!). With Sysbox, those outer containers are now properly isolated via the Linux user-namespace, truly enabling this use-case.

Ubuntu carries a few things that Sysbox relies on: a couple that come to mind are the shiftfs module (which Sysbox uses to enable the user-namespace in containers without requiring Docker to be set in userns-remap mode) and a kernel patch that allows overlayfs mounts from within a user-namespace (since the Docker running inside the container uses overlayfs mounts for its inner images). Having said this, we are looking at ways of overcoming these requirements to extend Sysbox to more distros; it's one of the most asked features.

There is plenty of info on Kubernetes (K8s) on the web, so I would start there. As far as running K8s inside Docker containers though, the use case would be one in which you want to run multiple isolated K8s clusters on a single host. One way is to use VMs, but recently people are resorting to using containers for this purpose due to their ease & efficiency. It's in the latter that Sysbox really helps, because it's capable of creating a container that runs K8s easily and with proper isolation. Typical use cases are testing, CI/CD, learning. But I would not discount this moving into production use cases in the future as the technology matures.

This is something we've not tried yet, so I don't know. I would be surprised if it worked right now. But in general, it's something we would definitely be interested in exploring.

The containers created by Sysbox act like virtual-hosts, so it makes sense to have the ability to expose GPUs / hw-accelerators within them. But container nesting, which comes naturally with Sysbox, would introduce another challenge since the GPUs would not just need to be passed to the outer system container, but also to the inner application containers.

We felt just providing enterprise support would not be sufficient to create a healthy business, given that Sysbox is designed to work under the covers (i.e., under docker/containerd) and does not require a lot of support. We opted for the open-core model as we felt it creates a good balance between contributing to the container ecosystem while still allowing us to sell some enterprise-level functionality (rather than just support).

Regarding the conflict of interest, we decided to handle it as follows: features that mainly benefit practitioners would go on the open-core, while features that mainly benefit enterprise deployments would not. Of course, there is still ambiguity there, but that will need to resolved on a feature by feature basis based on the feedback from practitioners and enterprises.

This is a learning process for us, but we understand this model is being used successfully by other IT infrastructure companies such as HashiCorp, so we opted for it.

Take a look at the Nestybox blog site: https://blog.nestybox.com/

It has an article on Docker sandboxing. Hope that helps!

Correct; Sysbox always enables the user-namespace in containers and manages the subuid/subgid allocation. In the open-source version, it assigns all containers the same subuid/subgid range, which is not ideal for cross-container isolation. In the enterprise-version (Sysbox-EE), it assigns an exclusive subuid/subgid range to each container automatically.

Thanks @lopezator.

That's funny ... brings back memories of the 90s :)

One thing I've noticed is that in modern IT infrastructure, there is usually two levels of sandboxing going on. At the low level you have VMs (sandboxed OS), and on top of it you have containers (sandboxed applications). Sysbox makes it easy replace that lower level with containers (which naturally leads to docker-in-docker or more accurately containers-in-containers).

To be clear I am not saying that containers are equivalent to VMs or that containers should always replace VMs. They are different beasts with different properties. But I am saying that in many scenarios it does make sense to use containers instead of VMs, particularly if your stack is all Linux, you don't need the isolation strength provided by VMs, and want the higher efficiency of the container.

It's hard to compare them because the goals are different. Kata containers seeks to harden container security by wrapping it with a highly optimized VM. Sysbox seeks to enable containers to run system-level workloads (systemd, dockerd, k8s) without requiring complex images, privileged containers, special mounts, etc.

Also, Sysbox is a pure OS-virtualization technology, which means it runs in environments where hardware virtualization is not available (e.g., a cloud VM, since most cloud providers don't allow nested virtualization).

Please take a look at these examples and let us know if they help:

https://github.com/nestybox/sysbox/blob/master/docs/quicksta...

Got it; note that inside a system container you can always run docker-in-docker using privileged containers (https://hub.docker.com/_/docker). That is, you don't need sysbox nesting in order to run more levels of docker nesting inside the system container. And those privileged containers would only be privileged within the system container, but not at host level.

Yes, as long as you can install Sysbox on the EC2 nodes, you are good to go.

Thanks, that's really encouraging. While it has taken a lot of hard-work to develop it, Sysbox would not exist without the excellent work done by OCI runc developers (Sysbox was forked from runc) as well as the LXD developers (who have done a bunch of the kernel work to enable the advanced OS-virtualization techniques we incorporated in Sysbox).

Got it; the answer is no, because I believe AWS ECS (Fargate) creates the containers using the OCI runc. In order for them to offer system-containers as a service, AWS ECS would need to run Sysbox on their backend to deploy the containers.

Great! ping us on our slack channel (the link is in the sysbox README file) in case you need help.

Unfortunately not. Sysbox requires "true root" privileges, so it can't run inside a system container deployed by Sysbox itself (since that system container would use the user namespace). You can run Sysbox inside a privileged container however, and in fact the Sysbox test framework relies on this heavily.

What use case do you envision for running sysbox in sysbox?

You can certainly install Sysbox on an AWS EC2 VM and launch system containers inside that VM.

Note that Sysbox currently requires Ubuntu Linux, because the latest Ubuntu releases use pretty new kernels and carry kernel patches that Sysbox relies on in order to perform some of the OS-virtualization in userland. See this doc for the distros supported by Sysbox:

https://github.com/nestybox/sysbox/blob/master/docs/distro-c...

We are actively working on adding support for more distros.