This is achieved by building inside of a chroot, with blocked network access etc. Only the dependencies that are explicitly listed in the derivation are available.
But you can achieve the same result if you use similar techniques with a bash script.
Just create a snapshot of the OS repo, so apt/dnf/opkg/ etc will all reproduce the same results.
Make sure _any_ scripts you call don't make web requests. If they do you have the validate the checksums of everything downloaded.
Have no way to be sure that npm/pip/cargo's package build scripts are not actually pulling down arbitrary content at build time.
You seem to be comparing 2 different things.
There are still ways that package will not be fully reproducible, for example if it uses rand() during build, Nix doesn't patch that, but stuff like that is fortunately not common.
Nix is the toolchain, which of course has its advantages.
Both are mostly running well-controlled shell commands and hashing their outputs, while tightly controlling what's visible to what processes in terms of the filesystem. The difference is that nix is just enough better at it that it's practical to rebase the whole ecosystem on top of it (what you refer to as a "toolchain") whereas Docker is slightly too limited to do this.
Using this approach you get something reproducible. Using apt-get in a docker file is an antipattern
We have 2-3 service updates a day from a dozen engineers working asynchronously — and we allow non-critical packages to float their versions. I’d say that successfully applies a fix/security patch/etc far, far more often than it breaks things.
Presumably we’re trying to minimize developer effort or maximize system reliability — and in my experience, having fresh packages does both.
So what’s the harm, precisely?
Regardless.... I can give a few reasons that it matters, off the top of my head:
1) Debugging: It can make debugging more difficult because you can't trace your dependencies back to the source files they came from. To make it concrete, imagine debugging a stack trace but once you trace it into the code for your dependency, the line numbers don't seem to make any sense.
2) Compliance: It's extremely difficult to audit what version of what dependency was running in what environment at what time
3) Update Reliability: If you are depending on mutable Docker tags or floating dependency installation within your Dockerfile, you may be surprised to discover that it's extremely inconsistent when dependency updates actually get picked up, as it is on the whim of Docker caching. Using a system that always does proper pinning makes it more deterministic as to when updates will roll out.
4) Large Version Drift: If you only work on a given project infrequently, you may be surprised to find that the difference between the cached versions of your mutably-referenced dependencies and the actual latest has gotten MUCH bigger than you expected. And there may be no way to make any fixes (even critical bugfixes) while staying on known-working dependencies.
nix provides the toolchain of reproducible artifacts... and then uses that toolchain to build a graph of content addressable inputs in order to produce a content addressable output.
So yes they are very different, but not in the way you are describing. Using nix, just like using docker, cannot guarantee a reproducible output. Reproducible outputs are dependent on inputs. If your inputs change (and inputs can even be a build timestamp you inject into a binary) then so does your output.
Building an image using nix solves many problems regarding not only reproducible environments that can be tested outside a container but also fully horizontal dependency management where each dependency gets a layer that's not stacked on one another like a typical apt/npm/cargo/pip command. And I don't have to reverse engineer the world just to see what files changed in the filesystem since everything has its place and has a systematic BOM.
And that all relies on discipline. Just like using a dynamically typed programming language can in theory have no type errors at run time, if you are careful enough.
FROM base-image@e70197813aa3b7c86586e6ecbbf0e18d2643dfc8a788aac79e8c906b9e2b0785
RUN pkg install foo=1.2.3 bar=2.3.4
RUN git clone https://some/source.git && cd source && git checkout f8b02f5809843d97553a1df02997a5896ba3c1c6
RUN gcc --reproducible-flags source/foo.c -o foo
but that's (IME) really rare; you're more likely to find `FROM debian:10` (which isn't too likely to change but is not pinned) and `RUN git clone -b v1.2.3 repo.git` (which is probably fixed but could change)...And then there's the Dockerfiles that just `RUN git clone repo.git` and run with whatever happened to be in the latest commit at the moment...
The only reason nix isn't dog slow is that it has really strong caching so it doesn't have to build everything from source.
You can have as many "FROM"'s as you want. "FROM scratch" along with "ADD" is also valid (for non-image dependencies).
From there you do not need to copy things, you can mount the reference into another stage directly.
Also, this is the Dockerfile format, the underlying build API's are _far_ more powerful than what Dockerfile exposes.
A Docker FROM is essentially the equivalent of a dependency in nix... but each RUN only has access to the stuff that comes from the FROM directly above it plus content that has been COPY-ed across (and COPY-ing destroys the ability to share data with the source of the COPY). For Docker to have a similar power to nix at building Docker images, you would need to be be able to union together an arbitrary number of FROM sources to create a composed filesystem.
People use the Dockerfile format because it is accessible. You can still use "docker build" with whatever format you want, or drive it completely via API where you have the full power of the system.
I'm not sure what you mean by 'You can still use "docker build" with whatever format you want'. As far as I'm aware, "docker build" can only build Dockerfiles.
I'm also not sure what you mean when you mention gaining extra abilities to make layered images via the API. As far as I can tell, the only way to make images from the API is to either run Dockerfiles or to freeze a running container's filesystem into an image.
Buildkit operates on "LLB", which would be equivalent to llvm IR. Dockerfile is a frontend. Buildkit has the Dockerfile frontend built in, but you can use your own frontend as well.
If you ever see "syntax=docker/dockerfile:1.6", as an example, this triggers buildkit to fire up a container with that image and uses that as the front end instead of the builtin Dockerfile frontend. Docker doesn't actually care what the format is.
Alternatively, you can access the same frontend api's from a client (which, technically, a frontend is just a client).
Frontends generate LLB which gets sent to the solver to execute.
Applying this information to the topic at hand:
Given what Buildkit actually does, I bet someone could create a compiler that does a decent job transforming nix "derivations", the underlying declarative format that the nix daemon uses to run builds, into these declarative Buildkit protobuf objects and run nix builds on Buildkit instead of the nix daemon. To make this concrete, we would be converting from something that looked like this: https://gist.github.com/clhodapp/5d378e452d1c4993a5e35cd043d.... So basically, run "bash" with those args and environment variables, with those derivations show below already built and their outputs made visible.
Once that exists, it should also be possible to create a frontend that consumes a list of nix "installables" (how you refer to specific concrete packages) and produces an oci image out of the nix package repository, without relying on the nix builder to actually run any of it.
This would subsume the purpose of e.g. https://nixery.dev/
Dockerfiles being at their core a set of instructions for producing a container image could of course be used to make a reproducible image. Although you'd have to be painfully verbose to ensure that you got the exact same output. You would actually likely need 2 files, the first being the build environment that the second actually get built in.
Or you could use Nix that is actually intended to do this and provides the necessary framework for reproducibility.
[1] https://en.wikipedia.org/wiki/Impact_driver#Manual_impact_dr...
It doesn't break as you scale. If you don't need that, then keep using Docker. (Personally, for me "scale" starts at "3 PC's in the home", so I eventually switched all of them to NixOS. I don't have time to babysit these computers.)
> Docker build are deterministic and easily reproductible
No, they definitely aren't. You don't really want to go down this rabbit hole, because at the end you realize Nix is still the simplest and most mature solution.
Docker build are deterministic and easily reproductible, you use a tagged image, that's it, it set in stone.
The 0.01% of Dockerfile that don't work, what does it even means, what does not work?
The other thing is about that buildGoModule module so now you need somehow a third party tool to use or build Go in a Docker image, when using a Dockerfile you just use regular Go commands such as go build and you know exactly what is going on and what args you use to build the binary.
As for the thing about using Ubuntu 18 which is out of date and not finding it, most orgs have docker image cache especially since docker hub closed access to large downloads, but more importantly there is a reason that's its not there anymore, it's not secure to use it, it's like wanting to use the JVM 6, you should not use something that is out of date security wise.