How does one build containers reproducably? I can make parts of my stack reproducable if I use existing, pinned containers, but due to all the `apt-get`ing in my dockerfiles, I'd call them anything but reproducable.

It's all relative, and as long as you don't use rolling-release distro, docker is a good compromise between having security fixes and partial reproducibility where it matters.

Remember, there were lots of containers before docker: jails, schroot, lxc, etc.. But the setup model there was "do the base os install, then enter the container and run setup commands" -- which is as non-reproducible as it gets, those commands are often not even saved anywhere. (Yes, you could if you are diligent about it. I don't think many people did that, certainly no tutorials mentioned it)

Compared to that, Dockerfile was revolutionary: a _requirement_ that all software is installed via automated means, and no simple way to do "just one more adjustment" in manual, undocumented way.

And then there was rkt around the same time as Docker and thought they would be able to take the mantle when Docker-the-company struggled to make money. The biggest problem with that theory is that it had none of this secret sauce you mention.

When I would point this out I usually got brigaded. Now RedHat owns the company that made rkt and I don't know if it's even being developed anymore. But nobody brings it up so I don't even have to think about it.

rkt (and many other container solutions) was introduced after docker was released and became popular... they even mentioned docker's shortcomings as a motivation for the project creation [0]. It had all the same problems as other replacement software: there were plenty of bugs and missing features (the announcement mentions "prototype quality release"), documentation was limited, and there are no community to help you. None of those would be fatal if it was significantly better than docker, but it was not -- it had less functionality and needed more scaffolding. So almost no one made the switch. It is closed now [1]

And why "rkt"? There were much better alternative container runtimes. For example Sylabs Singularity [2] -- container-as-a-file, instant mounting, etc... I wish more people knew about it.

[0] https://web.archive.org/web/20141201181834/https://coreos.co...

[1] https://github.com/rkt/rkt#warning-end-of-project-warning

[2] https://github.com/sylabs/singularity#singularityce

> but due to all the `apt-get`ing in my dockerfiles, I'd call them anything but reproducable.

Giving this some thought, I agree if you use package managers and base images I don't think it's possible to build images in a perfectly reproducible fashion.

But I would ask: don't image tagging and registries to provide a reproducible output (binary, packages, etc.) make it unnecessary to provide reproducible builds? Why does it matter if the build is perfectly reproducible if the actual output of the image can be reproduced? I.E. if I can run the software from an image built two years ago, does it matter if I can't build the exact Dockerfile from two years ago?

EDIT: fixed some grammar issues

I don't know, in my opinion then you are just running a container, like running an exe. You really aren't reproducing anything.

A lot of the value of containers seems to be in providing the ease of deployment benefits of a static executable to languages that don’t support compiling to a static executable.

Isn't that the whole point of a container is to not retain state of anything else within the filesystem other than the desired entrypoint/cmd(s)?

But that's the good thing about them. It isn't just a VM snapshot it is a recipe to build an image.

Pre container times there were recipes to build VMs including plain shell ( which can also be used to build containers).

Docker, the FOSs tools and the hub are very nice packaging.

Sure, but those were, and still are, a nightmare to use/learn for a lot of usecases.

that's where docker failed in only copying part of what I built (they simplified it, and to an extent that was good for making a product, but I had proposed the need for (and prototyped) a container/layer aware.

see https://www.usenix.org/conference/lisa11/improving-virtual-a....

the "problem" is building that layer aware linux distribution, so Docker just skipped it and relied on existing things and building each image up linearly.

This was actually an advantage of CoreOS's rkt (and the ACI format), where it enabled images to be constructed from "random" layers instead of linear layers (though the primary tooling didn't really provide it and were just docker analogues, there were other tools that enabled you to manually select which layer IDs should be part of an image - but docker "won" and we were left with its linearized layer model).

I use Nix, which has a `dockerTools.streamLayeredImage` function: https://nixos.org/manual/nixpkgs/stable/#sec-pkgs-dockerTool...

With apt2ostree[1] we use lockfiles to allow us to version control the exact versions that were used to build a container. This makes updating the versions explicit and controlled, and building the containers functionally reproducible - albeit not byte-for-byte reproducible.

[1]: https://github.com/stb-tester/apt2ostree#lockfiles

You use Docker as a distribution mechanism for the output of a reproducibility-oriented build system like Guix or Nix. Pin your dependencies inside the build system, then the build system can generate a container.

I use NixOS. But I see no reason to dabble with containers since I use NixOS except for when I can't coax some software to work reliably on NixOS or I want to produce binaries that work on distros that are not NixOS, in which case I can't use NixOS based containers anyway.

I had in mind that your goal with reproducible containers was to distribute some app your business makes on platforms that don't know anything about NixOS, in which case building the app with Nix and then using your favorite container runtime as the deployment target makes sense. But if you're using containers as an escape hatch for NixOS and Nixpkgs, that definitely doesn't do much for you!

In enterprise environments, some old school distros have an archiving layer that sits between hosts and their normal repos that you can use to hold back updates. Maybe you could use something like that. I forget what Red Hat's offering is called but I think it's part of Satellite. Idk if there are any free tools for that, but maybe there are.

The other alternative escape hatched that NixOS has, like FHSUserEnvs or just steam-run, you likely already know about.

> I can't use NixOS based containers anyway

I'm no expert, but I believe the purpose of containers is to include all user space dependencies... so this doesn't make sense to me.

I personally am surprised NixOS hasn't leaned into marketing itself as ideal for creating reproducible containers.

If you're using NixOS on the desktop, sometimes you want to run an obscure piece of software quickly, without packaging it for NixOS, which can be a PITA with proprietary software or software with irregular build processes.

Or maybe you're working in an organization where some instructions are written for another distro, and you just want to be able to follow them word for word the first time you attempt a task, to make sure you understand the process on a 'normal' distro, or because you're troubleshooting with someone who is running another distro. Then NixOS' support for running containers is handy, but afterward you're left running some containers whose reproducibility doesn't match what you've come to expect frok the rest of your system.

> I personally am surprised NixOS hasn't leaned into marketing itself as ideal for creating reproducible containers.

Agreed, I think this is a really good use case for a lot of companies.

Its great when it works, but I use containers to emulate normal Linux when it doesn't.

How reproducible is up to you.

You can build everything elsewhere and only copy in the artifacts. Or only install pinned versions of things. Or only use certain base tags. Or do the entire CI thing inside a container. Or use multi-stage containers for different steps.

My previous startups all used pinned based images with multi-stage containers to build each language/framework part in its own step, then assembly all the artifacts into a single deployable unit.

The Docker images are not even versioned.

I've seen success building them in centralized server(s) and 'apt-get'ing specific dependencies instead of the latest ones.

I don't think it was even that. It was massive marketing and hype which drove docker adoption. The engineering concerns and the user experience were both negatives because they actually added a considerable cost and friction to the experience.

I will obviously need to back that comment up but fundamentally the security posture of docker (and most package repositories) is scary and the ability to debug containers and manage them adds considerable complexity. Not to mention the half baked me-too offerings that circle around the product (windows containers anyone?)

The repeatability argument is dead when you have no idea what or who the hell your base layers came from.

What it did was build a new market for ancillary products which is why the marketing was accelerated. It made money. Quickly.