Changing the base layer itself at a minimum requires people to upgrade, and now you don't have that advantage of the preexisting audience anymore. If your improvement requires a breaking change, then you're in a real pickle. So people will stack on more and more until it becomes more or less unbearable.

However, a minor improvement or bugfix should be done in the component that is responsible for it. Creating another layer instead of fixing the problem is just creating more problems.

But, unless you are willing to abandon Linux/Unix, this is kinda where you are left.

Common Lisp is my go to example of this. It's possible to extend the language to support entirely new paradigms without breaking it, because of the power of the macro system.

You can't take an AWS VM and fire it up on DigitalOcean without trial by fire. VM's should be portable but they're not because cloud providers don't want them to be.

For a while everyone was locked into AWS but now that there's other options some companies want to hedge their bets, or even run parts of their workload locally.

Since the VM space is all proprietary now people are moving one level up and building a consistent environment above it. I fully expect docker to become proprietary in a couple years and the cycle will begin again.

You can't ?

I have never used DO, but I have deployed linux and FreeBSD systems on EC2 and moving them to bare metal was just a tar command away ... you can even pipeline 'dd' over ssh if you want to be fancy ...

Modern package management systems like APT spend a lot of effort installing and removing files, and they don't do it completely; any file created by a program after it is installed will not be tracked.

You could accomplish the same thing in other ways (as Apple's sandboxing tech does), of course.

Well, there /is/ another way to do it.

STATIC LINK ALL THE THINGS

Which would work if licenses and copyrights didn't exist.

We're at the point in the hype cycle where it starts getting fashionable to dismiss that as an overkill, but the reality for most of us out there is that most software is way more complicated than a single executable and containers make it easier to deploy complicated software.

Managing internal dependencies (like libraries) is another concern entirely. But containers are good for that, too.

I don't think it would.

Dynamic linking allows a library to be patched once and have the patch apply to all the programs using it. If every program was statically linked, you would have to update each one individually.

Not to mention the waste of space.

I'm guessing much of that is moot these days, but IMHO it's still something to aim for.

The benefit of that goes away with containers anyway, you don't share libraries, every instance gets its own install.

On the other hand, deploying stuff is dead easy now. You just tell the hoster "deploy this Docker package, expose port X as HTTP, and put a SSL offloader in the front" and that was it, no more "we need <insert long list> to deploy this" or countless hours spent with the hoster on how to get weird-framework-x to behave correctly.

If it's actually useful we'll find someplace for it to live (or just in containers if that's all that's needed). If not, finding that out was cheap.

I just sometimes wonder if the cart hasn't gotten in front of the horse on the whole container front. The fact that the term "bare iron" has been hijacked to mean "not under virtualization" made me start to think that something is seriously fucked up.

I don't like Docker, but I think it does some differential compression with the layers when you are modifying the image. So you don't have to re-do this install from scratch.

You may also run into issues with user IDs and various system config files in the chroot. Configuring the service with flags, env vars, and config files is a bit of a pain.

Docker is essentially a glorified chroot... I've essentially tried to rebuild it, and it is unfortunately a lot of work.

Heads up, you're talking about a different thing. You want to run these things as normal operations, but gunnihinn was talking about deploying an application to check if it is of any value. Chroot is just enough to make a mess as the application's developer instructed in INSTALL.txt without the need to worry about cleaning up afterwards.

And by the way, you seem to be confusing containers and Docker.

And I'm not confusing containers and Docker, I'm just speaking a bit imprecisely. In my experience, conversations about "containers" are rarely about raw containers, but rather some specific containerization scheme and tools (e.g. docker). I suspect everyone in this entire subthread means "docker containers" when they says "containers."

Yeah, some newbie forgot to mount-bind a necessary directory like /proc or /dev, didn't provide sensible /etc/resolv.conf, or messed up host's and chroot's paths, either in request or in configuration. Nothing that would render chroot unviable. Is this what you meant?

There are several capabilities in that list to address your problem. Since your question is concerned about memory you can search for "memory" in that page and note the exact capabilities you will need.

OpenBSD has even more features (https://en.wikipedia.org/wiki/OpenBSD_security_features) alongside the capability model.

My gripe is that instead of learning how to properly use their tools and platforms people have started looking for golden hammers like docker and now we have a mess like RancherOS. If someone can explain to me how isolating the system services in docker containers (which by the way are not isolated since they are privileged) does anything above and beyond what capabilities and the security features in OpenBSD provide then I'll concede the point. My guess is there are no advantages and people are jumping on bandwagons they know nothing about and since the previous generation of tools was not utilized to its full extent neither will all the new docker hotness. People are hammering square pegs into round holes.

My point was more about the "developer UX" of needing to isolate things that way. Containers have the semantics of isolating a group of processes, but not performing any internal isolation between the processes in the container. This is almost always what you want—you want "your app" to be able to have multiple processes, and to not have any security boundary between the parts of "your app", just between "your app" and "other apps" or the OS. In other words, you want to have a single "process"—from the OS security subsystem's perspective—whose threads happens to be composed of multiple PIDs, multiple binaries, and multiple virtual-memory mappings. You want to fork(2) + exec(2) without creating a new security context in the process.

Sandboxing would make perfect sense if single processes were always the granularity that "apps" existed at. Sometimes they are. Sometimes they're not, and people do complex things with IPC-capability-objects. And sometimes, when they're not, that fact pushes people toward avoiding multi-process architectures in favor of monolithic apps that reimplement functionality that already exists in some other program in themselves, in order to bring that functionality into their platform/runtime so it can live in their process.

Containers let people avoid this decision, by just applying things like capabilities at the container level, rather than at the process level.

> If someone can explain to me how isolating the system services in docker containers (which by the way are not isolated since they are privileged) does anything above and beyond what capabilities and the security features in OpenBSD provide then I'll concede the point.

Completely apart from the above, my understanding of Rancher is that it's the Docker part of "Docker container", not the container part, that provides the benefit there. Docker is a packaging and service-management system; that its packages use containers is frequently beside the point. Rancher's system services are Docker images (i.e. Docker "packages"), and so you use Docker tooling to create, distribute, manage and upgrade them. If your own application on such a system is managed through Docker, this provides a neat solution to unifying your operations—you just do everything through the docker(1) command.

That is a complex mix of services running on a machine, some sharing the same flavor of VM, and you're allocating fractions of the total available resource capability to different components. If you cannot make hard allocations that are enforced by the kernel you cannot accurately reason about how the system will perform under load, and if one of your missions is to get your total system utilization to be quite high, you have to run things at the edge.

Agreed. What I find so odd is that this problem was simply and elegantly solved in BSD with 'jail' and everyone went about their business.

I do not understand why (what appears to be) the linux answer to 'jail' is so complicated and fraught and the subject of so much discussion.

I am not sure that containers and their build scripts represent the $huge_profit_potential that people think they do ...

I think there's interesting ideas of environments in other operating systems, but I'm not seeing how you'd make things better by flattening containers into the OS, per se.

Making existing programs compatible with such a paradigm probably wouldn't be any more work than e.g. adding SELinux/AppArmor support.

Now of course the correct way to solve this would be (a) to make both programs use the same version of python-somelib, (b) make upgrades happen atomically, (c) for Python modules to actually have some API backwards compatibility. But in the absence of doing the right thing you can use a container to effectively static-link everything instead. And worry about the security/bloat/management another time.

The main advantage of docker, as far as my understanding goes, is more of a prebuilt system configuration thing. Need a database? Load the prebuilt PostgreSQL image onto the respective machine.

One thing I think should get more use in general is that Dockerfiles are essentially completely reproducible scripts[1]. Too many companies I've seen still use Word documents full of manual steps one can easily get wrong for all their machine setups (especially in the Windows world). If you want to test something quickly, you're bogged down for a day.

[1]: example: https://github.com/kstaken/dockerfile-examples/blob/master/m...

Now it's all docker rather than packages, ansible (which leaves no trace of what it's doing on the target machine) rather than a for I in 'cat hosts'. Fine, but where's the benefit?

I can see, however, a benefit in encapsulating different services in different containers, as this potentially gives you some control over them (available disk space, network usage etc.) * . On top of that, I imagine starting out on a "machine agnostic" approach can be rather useful if you have to change your network landscape further down the line: If your database already is configured as if it were running on its own server, there's certainly a bunch of unintended coupling effects you can avoid.

That said, I can't see Docker being the silver bullet it gets hyped up to be sometimes. But that goes for most new and shiny things in the tech space...

* And yes, that's already feasible without containers. Docker's approach to this seems to provide a way to do it in a much more automated way than most alternatives though.

* https://lists.debian.org/debian-user/2017/04/msg00151.html

* http://www.panix.com/~rst/tuttle.html

* https://lists.debian.org/debian-user/2017/04/msg00437.html

* https://github.com/rst/Tuttle

I think the main advantage is that it standardizes the interface around the application image/container. This allows powerful abstractions to be written once rather than requiring a bespoke implementation for each way that the application is structured. Imagine writing the equivalent of Kubernetes around some hacked-together allows-multiple-versions-of-a-dependency solution. It would be a nightmare. But because the Docker image/container interfaces are codified, you can build powerful logic around those boundaries without needing to understand what's inside those images/containers. Dynamically shifting load, recovering from failures, automatic deployments and scaling are all much easier when you don't have to worry about what language the application is written in or how the application is structured.

The point isn't really to "run more Docker". It's to eliminate as much operating system overhead as possible, so that nearly every CPU cycle and byte of memory usage is dedicated to your containers.

For example why run a network supervision daemon if dockerd or equivalent handle all the important complex pieces of networking via container orchestration? Why have a local package manager, or system port mapper.

Personally I prefer running SmartOS and Triton containers. Their system seem much more stable than any of the Linux containers and/or systemd setups I've tried. It sticks a bit more to traditional unix design which makes sense to me. Items like the caching layer for containers build on ZFS snapshots, a well tested file system layer, rather than ad hoc userland tools. Triton also runs all of the orchestration layers in separate zones (containers) like RancherOS is trying to build. But each component is a simple(ish) service, it's easy to `zfs list` and check on a container's file system or fix it or backup, etc. Same with SVC or VM machine management which both have small simple tools that do one thing pretty well.

To that note, docker has been moving towards breaking out and using smaller daemons haven't they? If that continues it might turn out more modular in the end wherein RancherOS would end up being more modular than systemd Linux setups. Imho, that'd be great.

By all means continue to run RancherOS and let me know how that goes when you're managing a few hundred to maybe upwards of a thousand VMs and then layering a container orchestration system with the underlying VMs coming and going on an on-demand basis. I remember thinking "I really wish I had more of this docker stuff in the OS itself. Because dealing with all the caching, volume mounting, and instability in userspace is so much fun".

I'll await your report because clearly my experience with these systems and all the ways they fail is too combative for your taste. There are a few things they don't tell you on the brochure when you're drinking the kool-aid.

I might give it a crack based on that. If you are hell bent on running Docker then an equivalent of the Ubuntu minimal install seems a good way to start.

If this idea has similar pragmatic advantages over the would-be best solution, then I'm game.

yep, when you want a distributed filesystem building it on top of a good and already working local filesystem is a pretty robust and cheap approach.

>smell like a system on top of a system to fix something that could be fixed in the system.

container layer is basically a distributed OS. Making a distributed OS by "fixing in the system" is pretty much non-starter ... or a huge academia project.