Containers halfway solved some big existing problems that most people don't seem to see very well.

Packaging is hard, and both debian-based and rpm-based (and really most other's I've seen) are pretty awful. (except BSDs, which I've had a lovely time with)

They're slow, they're stateful, writing them involves eldritch magic and a lot of boilerplate, and they're just frequently broken. Unless you're installing an entire OS from scratch you're probably going to have a hard time getting your system into the same state as somebody else's. And running that from-scratch OS install is definitely possible in a as-code way, it can take an hour.

Containers came along and provided a host of things traditional packaging systems didn't and they took over by storm and with them came a whole lot of probably unnecessary complexity from people wanting to add things. Adding things without ending up with a huge mass of complexity is hard and takes a lot of context knowledge.

So we ended up solving a host of problems with containers and creating a whole new set along the way.

(Almost) nobody is using Arch Linux on servers, but I find its package system to be very good (not surprising since it was mostly copied from BSD ports).

A few random examples (not the best you could find, just something I've used recently):

- re-packaging pre-built binaries:

https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=visua...

https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=nomad...

- building C from source

https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=tinc-...

- building Go from source

https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=yay

- patching and building a kernel

https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=linux...

Alpine (apk) and Void (xbps) have similarly nice packaging systems.

I was having problems building wine. So I used the arch pkgbuild and just didn't do the install phase. Made compiling pretty simple. And all the outputs are nicely defined in the aur repo locally.

Does Arch support installing multiple versions of libraries?

Yes, but only if they're packaged separately. PKGBUILD is easy, so it takes very little effort to repackage older library versions under a new name (and patch dependents to use the new name) if you need them.

Not really. There are a few applications that can be installed because the install path different for each version, things like Java support this. But libraries like glibc, etc. are tied to one version so anytime those libraries change it triggers a rebuild of many packages.

> Containers halfway solved some big existing problems that most people don't seem to see very well

A big reason for that in the past much fewer developers were confronted with this problem domain.

In larger companies packaging and deployment was often the responsibility of ops, with some input from and interaction with development. That of course also meant much longer lead times, arguments about upgrading versions of libraries or other executable dependencies, divergence of production and development/test environments, and the associated unfamiliarity with the production environment for developers and hence often more difficult debugging.

Ever since Docker (+ Kubernetes and various cloud specific container solutions) became so popular, a lot of devs now at least partially deal with this on a regular basis.

Which is mostly a good thing, due to the negatives above.

> Ever since Docker (+ Kubernetes and various cloud specific container solutions) became so popular, a lot of devs now at least partially deal with this on a regular basis.

But that’s in line with the whole premise of DevOps, right? That the strict separation between dev and ops is a bad thing, and it’s good that devs get involved with ops and vice versa.

I don’t think this has to do with containers per se, but they do help a lot with that goal.

Agreed. The core concept is that we should automate away as much of the ops workload as possible so (1) devs don’t need to learn the whole ops skill set and (2) no one is doing things that computers could do automatically. Containers and orchestration technologies are a form of automating away a lot of ops work (if you need to package an application you don’t need to solve for SSH, package management, log exfiltration, monitoring, or any of a dozen other things).

I think those are very good points. In my opinion, the hoisting of packaging concerns from language-level and OS level was inevitable, and containers an an okay way to do that.

Your experience seems to be the exact opposite of mine - debian packages are typically very elegant.

Most corporate use of Docker I've encountered is a mess of stupid patterns like "RUN command A && command B && command C && ..." to reduce layers or some such nonsense which makes debugging build failures tedious.

> They're just the symptom of the disease that is future shock.

Yes, absolutely, and I hope you mean that in the capital-F "Future Shock", Alvin Toffler sense, because there is a lot he wrote that hasn't even been carried over and digested. Software is an endlessly disorienting sea of change, getting faster and thus worse as time progresses, and it's frankly madness at this point.

It seems absolutely no one is committed to providing a stable platform for any purpose whatsoever. Even Java, where I spent many years being ingrained with the absolute necessity of backwards compatibility with old (perhaps even dumb) classfile versions, has been making breaking changes as part of its ramp up to semi-annual major version releases. Node Long Term Support "typically guarantees that critical bugs will be fixed for a total of 30 months."[1] Pfft. It's a joke. You can't get your damn API design straight by version 12? I'll do my damnedest to avoid you forever, then. It's so unserious and frankly irresponsible to break so much stuff so often.

But change only begets more change. We're all on an endless treadmill, constantly adapting to the change for no reason. And people have to adapt to our changes, and so it goes.

[1] https://nodejs.org/en/about/releases/

That’s ingenious to write off Java as not backwards compatible. The only change they did was closing the doors to the internals of the JVM because those are implementation-dependent anyway (meaning those programs wouldn’t have worked on anything not OpenJDK) and are likely to change. You can still absolutely run a class file compiled to Java 1.0 on a JDK 16.

How about Golang in this case? AFAIK there haven't been any breaking changes yet.

It is definitely a breath of fresh air in that regard.

This is well put.

Containers side-stepped the deficiencies of Linux distributions, which had become so based on 'singleton' concepts; one init system, one version of that library etc.

A shame because there's an inherent hierarchy; everything from the filesystem, UIDs, $LD_LIBRARY_PATH that could really allow things to co-exist without kludge of overlay filesystems. Just it was never practical to eg. install an RPM in a subdirectory and use it.

Containers aren't a very _good_ solution, they're just just best we've got; and still propped up by an ever-growing Linux kernel API as the only stable API in the stack...

This sounds like a lot of stuff that Nix solves, the multiple versions of libraries coexisting part at least

Nix strikes me as the Linux community looking at an overly complicated problem of their own making and deciding that the solution is to add even more complexity.

Don't get me wrong, from what I hear Nix actually does deliver on the promise for the most part, it's just that you have to learn a new language to use it effectively and of course it has its own quirks.

What solution wouldn't require its own specification + quirks? Whether its a Dockerfile or nix package I don't see the difference besides people tend to be familiar with only 1 of the many options.

Im not comparing whether Dockerfiles or buildpacks or nix packages are more ergonomic than one another but i do think your comment is...misguided. From what I have heard Nix is pretty wonderful to use and simplifies the problem - it just requires you learn about Nix a bit which i think is a fair trade-off for the benefits it supposedly provides

Anyone can build their first Dockerfile and deploy it for Node.js or similar in like 5 minutes. The tricks that make images smaller later use the same concepts and syntax. There is a reason it took over the world so quickly.

Nix ... I have so far spent about 10 hours learning it to manage my machine. I have forgotten about 98% of it and abandoned the project. You feel like you're sitting in the middle of a spider web, and you can sense the whole system at once. Literally none of your prior knowledge of how to use a computer will help you. None of your existing build tool CLI can be used. Every package manager needs a nix-ifier, like node2nix. Everything you see in a nix file will have to be googled, searched in the documentation, searched in GitHub repos for some kind of example. Nix has rebuilt the world from scratch.

If you're trying to make the next big thing, try to make it leverage people's existing knowledge. One truly excellent example is `compile_commands.json`. It does a very similar thing to Docker, where it extracts information from your existing build process, without actually changing the build process. The problem statement was that people wanted LSP (and predecessors) implementations to have access to a list of input files to a C/C++ compiler, but they didn't want to abandon Make and CMake etc. So they basically made a structured log of all the CC invocations, and a wrapper around CC that would parse the arguments and write to the log in JSON format. These days you get it for free with CMake[0]. You can use it with nearly every C/C++ build system on earth with a single CC=... argument to make.

[0]: https://cmake.org/cmake/help/latest/variable/CMAKE_EXPORT_CO...

Your compile_commands.json example is essentially what tools like node2nix produce. For saner build systems, e.g. cmake and meson, simply including the build tool package in `nativeBuildInputs` along with dependent packages in `buildInputs` suffices to build 90% of C/C++ packages with Nix, with the remaining 10% being Qt (I jest).

The real killer app for Nix is in testing now, and that's the "flakes" feature. Lots of this stuff will get way easier to use when you can throw "github:owner/repo" in an `inputs` set and get a working Nix builder for your project without needing to read through nixpkgs. I hope you give it a try again sometime, as it has changed my perspective on how software should be built, distributed, trusted, and deployed.

Nix is still trying to be a thing that replaces other build tools. The generated nix file is a recipe for doing what “npm publish” does when it builds a tgz. Every single language has a different way of doing this, and will require its own *2nix to be maintained alongside changes to eg package.json, Cargo.toml, Package.swift. The 2nix bit is the hard part; every package manager has to be re-implemented.

Docker does not have this problem at all. Every build tool on earth works with it, with zero configuration.

Compile_commands.json may have similar output to node2nix etc, but it infects nothing, replaces nothing. It works with all the different makefile alternatives with no additional effort. The closest similarity is with Docker: Docker builders intercept at the file system layer, covering every build system ever; compile_commands at the standard GCC-compatible shell arguments layer, covering ~all C build systems and compilers. Nix does not intercept anywhere, it asks you to use a new tool to do everything from scratch in a Nix-compatible way, covering no build systems.

That’s not to say it isn’t great when you have already built a Nixified package manager replacement for your specific language ecosystem. But it’s not going to take over the world like Docker and compile_commands did. Imploring people to give it a shot is the only way, unfortunately. I will remain open to it, especially if someone can figure out a force-multiplier for these 2nix implementations.

> Nix is still trying to be a thing that replaces other build tools.

Not true! Nix wraps other build tools, and provides hermetic and reproducible environments to those tools. If the tools exposed a way to get the URL and SHA256 hash of every dependency it downloads from the Internet, then the "infection" doesn't need to happen, as you would simply supply those hashes to Nix, which in turn will happily allow them to be downloaded in the sandbox by the tool. That tools like node2nix exist speaks to the walled garden created by these tools and ecosystems, because they do not (easily) expose dependencies to their environment, and/or they do not (easily) accept dependencies from their environment.

This would absolutely be a problem with Docker as well, if you added the same requirements that Nix enforces in its sandbox, because otherwise you are allowing Docker to fetch dependencies by URL without specifying their contents.

> If the tools exposed a way to get the URL and SHA256 hash of every dependency it downloads from the Internet, then the "infection" doesn't need to happen

Yes. Good start. If you can make it so that exposing this information to Nix is easy enough that e.g. the NPM team does not need a PhD in Dhall to write it to a file, then Nix will be a much more solid proposition. That data alone isn't enough, but that + a DAG of what NPM will do to the downloaded tgzs is much closer. It's also enough for cargo. And many other languages. Dhall is cool to write by hand but, back to my original example, compile_commands.json could be written by a monkey. It needs to be that easy. It needs to be as easy as printing GraphViz DOT to stderr. Then and probably only then will Nix support start getting upstreamed.

Dhall is probably Nix's biggest liability at the moment; they sought to make a single language, with a rapidly changing API, for configuring your computer (by hand) as for making compilers reproducible. Compiler output! In an essentially esoteric configuration/programming language, which takes a lot of effort to port to a new ecosystem! No. Use JSON. Ideally you will never have to actually write Nix, the same way humans have never had to write compile_commands.json by hand, and the way nobody has ever had to construct a Docker image by hand out of individual tar files.

Meanwhile, NixOS forces me to configure systemd units by writing Nix, not by writing systemd unit files.

Re force multipliers, I think you could do much worse than an HTTP proxy! Many package managers support folks behind corporate firewalls, and you can then support any thing that fetches code from anywhere else. Give it a specific Node.js version and a proxy, run npm install, log all the network requests into a nix file. Biggest issue is TLS probably.

You can’t really leverage what is fundamentally broken. To live up to Nix’s claims it has to heavily patch software. But this is a self-fulfilling thing, once it caters to enough people, other tools will natively support nix.

You're right that Docker is similar to Nix, at least in the sense they both seem to be trying to work around problems with the Linux packaging and library ecosystem by piling more of their own complexity on top. I suspect the comment you replied to wants to see the actual underlying problem solved.

To use an example from another community, no amount of performance improvements to NPM will ever make it a good idea to depend on hundreds of one-liner "is number odd" or "left pad" packages. Papering over the problem with yet more technology only ossifies it, making it harder to solve for real.

Yes, no doubt there's been some solutions within Linux distributions that think more outside the box (though I'm not familiar with Nix). There's many home-grown solutions in production environments within orgnisations, too.

As you suggest, these are probably 'pieces' of the puzzle, by no means 100% identical to how containers are used today. But I think we'd have ended up in a different place.

My understanding though is that nix tries to solve this globally (it manages your whole system, or your whole home directory, as opposed to docker, which has clearly demarcated separation between different images), and it doesn't reuse existing packaging (in particular the language, as in "apt install" etc.)

There's definitely advantages that way, but there's also drawbacks.

There's NixOS, which you are describing, and then there's `nix` the package manager and build system. The latter is installable on any Linux or macOS system as a normal binary, and can be used standalone as a build tool. My company uses this to standardize (and cache!) development environments across a diverse set of hardware; the environments do double-duty as hermetic CI builders as well.

> getting devs to actually care if their software runs on platforms more than a year old.

This is why we don't play games with siloing responsibilities on the tech stack. Every single developer on the team is responsible for making the entire product work on whatever machine it is intended to work on. No one gets to play "not my job", so they are encouraged to select robust solutions lest they be paged to resolve their own mess in the future.

Maybe those solutions are containers in some cases, but not for our shop right now. Our product ships as a single .NET binary that can run on any x86 machine supported by the runtime.

So you don't support M1 Macs ?

No - the part of the product implicitly discussed above does not. We don't really have any intentions of running our services on piles of macbooks at the moment.

That said, we do have an iOS client which is intended to run on such classes of devices. I loathe the fucker so much (dev experience is garbage) but our customers like it a lot so... here we are. 99% of the complexity lives on the server, so the app is not a daily struggle. We also have a UWP client, but it has its own set of "difficulties" that I won't get into at the moment.

At some point I want to try to build a pure HTML5/canvas solution that can be served from a cheap-ass linux box and consumed by any device with a reasonable web browser implementation.

Looks like .Net 6 (formerly Core), due for full release in a couple months supports M1 Macs[1] as a build target just fine. So does MAUI[2]

1. https://github.com/dotnet/runtime/issues/43313

2. https://docs.microsoft.com/en-us/dotnet/maui/get-started/ins...

M1s support x86 via emulation.

Containers solve the problem of clashing library versions needed by different applications running on a single host (and I know there are other ways to solve this).

This is really not a new problem :) I remeber dealing with shared libary versioning issues from no long after I started in IT in the 90's and it's been a problem since.

Solving that problem seems like a win to me.

Containers add a substantial level of indirection for us, developers. Now we have to grow a seventh arm to juggle to manage/fold into our workflow. For production? Hands down the right solution. For development, I wish, only wish, We could live without.

It's made the entire process MUCH easier for me... `docker-compose up -d deps` and I have all my background services running local... `... api` and the api is also running and I can concentrate on the UI.. `... ui` and it's all running. I can then run tests against the whole thing.

Also, setup all the containers to include unit test results in the runtime container... this gets extracted/merged in CI/CD. Beyond this, I can stand-up the entire application and run through full integration and UI test suites in the CI/CD pipeline. Same commands locally... it all is much smoother than prior experiences.

I will NEVER run a database install on my developer desktop again. Database deployes on the main application I work with, and unit tests all finish in about 5 seconds or less (not including initial download). I'm also able to run db admin apps right with the DB.

Persist volumes, run/test upgrades and from-scratch. It all goes really smoothly overall. Wouldn't ever want to go back to mile-long dependency instructions step by step to getting a development environment running ever again. WSL2 + Docker Desktop are pretty damned great.

its solved until your company installs a CVE scanning tool such as twistlock and it turns out you have 100 different docker images out in the wild you need to update, rebuild, and re deploy.

Containers aren’t the final destination, but they’ve enabled polyglot orchestration i.e., an app developer can target Kubernetes without needing to manage the minutia of operating a bunch of Linux hosts. It seems like almost every company that isn’t using containers for SaaS software development ends up badly reinventing Kubernetes and sinking a ton of time and money into maintaining it, and as a “human person”, I’m glad that I can focus my efforts on higher-level problems. When a technology inevitably matures to replace containers, I’ll look into it, but for now containers are the best way to build and manage heterogeneous distributed systems.

For that matter, pushing for SRE roles that manage orchestrating the K8s environment and as a developer you can focus on a local docker-compose, and spend more time in testing (unit and integration). The Developer is responsible for Dockerfile, and the CI/CD build and test portions of the process.

Considering the level of options from Kubernetes, heml, istio, etc can get complex, the developer can focus on the boundary requirements... expected environment variables and peer systems/services.

I think that’s precisely the opposite of the SRE/DevOps model. The developers shouldn’t be managing their own clusters (istio, etc) but they should be able to define and maintain their own applications (not just the container code but also supporting infra).

Assuming that I can now take away time from other things to gain the knowledge to do so, sure. Also, taking away the development time from all other developers to do so.

I don't think dependencies is the only benefit of containers. I personally like the isolation they provide and generally prefer running services in containers, even if they are using the same dependencies as my OS. I run Linux too, so I don't have to worry about any virtualization framework overhead.

Namespacing and isolation also unlocks additional features, such as VM-style checkpointing and migration (via the CRIU featureset, which AIUI is now part of the mainline kernel). Moreover, the 'container' workflow provides a common interface that the various sorts of orchestration/deployment/management platforms can then rely on.

> I personally like the isolation they provide and generally prefer running services in containers, even if they are using the same dependencies as my OS.

I would also not downplay the importance of Docker's support for software-defined networks and it's ability to arbitrarily configure networking at the container level.

I firmly believe that networking doesn't pop up so often while discussing Docker because Docker solves that problem so fantastically well that a complex problem simply ceases to exist and completely abandons everyone's mental model.

Having to define complex networking completely internal to a server is a problem that docker created, not one they solved.

Have they fixed ipv6 support yet?

Yes they have, but there is also a plethora of other container daemons and tools now aside from Docker with IPv6 support.

Java running in a container is somewhat amusing because of this. So you have a several solutions to the problem of agnostic packaging (java/jar/ear/war/etc) running inside another whole solution for agnostic packaging.

That’s just Java EE (now Jakarta) and Spring (when deployed to an application server). And indeed, these were containers before they were cool, with DB connections, pooling, it can connect to messaging services, etc. plus gives an environment where classes can easily do dependency injection, transactions, concurrency and many other things.

I don't think there's another way to ship custom certificate authorities without using containers? It's something you absolutely have to do around here if you want to interact with government APIs of any kind.

I relatively rarely work with Java and am probably mistaken.

I'm not saying containers aren't needed. Just that we keep trying to solve packaging and end up with more layers that have to duplicate large swaths of functionality. So we get java->containers->container orchestration, for example. The containers overlap some built-in java functionality, and so does the orchestration piece.

Nix (and guix) does solve this issue novelly, and for some reason it doesn’t get the recognition it rightfully deserves.

The problem doesn’t start with virtualization, that is indeed a side-track.

Containers just moved the compatibility barrier up the abstraction stack. That’s not terrible (fewer and fewer understand how their computer actually works) but all those same problems still remain. Now they just apply to remote APIs instead

But I don't want to have to care about any of that stuff and containers let's us not care about it. That's a huge solve.