Actually, it's Debian guys who pushed reproducible build hard in the early days. They upstreamed necessary changes and also spread the concept itself. This is a two-decade long community effort.
In turn, NixOS is mostly just wrapping those projects with their own tooling, literally a cherry on the top. NixOS is disproportionately credited here.
Though you can and should sign the mapping!
I mean it's worthwhile to fix, but that behaviour seems so standard.
It is really, really annoying that the Rust team is not taking this problem seriously.
Obviously the binary can't contain paths from the execution machine because it doesn't know what the execution machine will be at compile time, and the source code isn't stored on the execution machine anyway. The point of including the source path in the debug info is for the developer to locate the code responsible if there's a crash.
So is Nix/NixOS, which has reproducibility in mind from the start.
The earliest example I can find is "Nix: A Safe and Policy-Free System for Software Deployment" from 2004 ( https://www.usenix.org/legacy/event/lisa04/tech/full_papers/... ):
> Build farms are also important for release management - the production of software releases - which must be an automatic process to ensure reproducibility of releases, which is in turn important for software maintenance and support.
Eelco's thesis (from 2006) also has this as the first bullet-point in its conclusion:
> The purely functional deployment model implemented in Nix and the cryptographic hashing scheme of the Nix store in particular give us important features that are lacking in most deployment systems, such as complete dependencies, complete deployment, side-by-side deployment, atomic upgrades and rollbacks, transparent source/binary deployment and reproducibility (see Section 1.5).
Despite the fact that Debian (as a project) has shouldered far more of the work with upstream projects to make bit-identical reproducibility possible at build time, Debian (as a distro) doesn't have a design that makes this kind of reproducibility as feasible, practical, or robust at the level of a whole system or disk image in the way that NixOS has achieved here. To quote the Debian project itself[0]:
> Reproducible builds of Debian as a whole is still not a reality, though individual reproducible builds of packages are possible and being done. So while we are making very good progress, it is a stretch to say that Debian is reproducible.
Beyond the fact that some packages still have issues upstream and the basic technical problem of versioning (i.e., apt fetching binaries from online archives in a stateful way) Debian additionally struggles with an extremely heterogeneous and manual process of acquiring and uploading source packages[1]. Debian doesn't even have the resources to construct a disk image where the version of every package is pinned, short of archiving all the binaries (which is how they do ‘reproducible’ ISO production now[2]). But pulling down all of the pre-built binaries for your distro isn't really ‘reproduction’ in the same sense as ‘reproduction’ in Debian's (package-level) reproducibility project.
Some points of comparison
• NixOS always fixes the whole dependency tree
• Debian requires a ‘snapshot’ repository to fix a dependency tree
• most NixOS packages are updated through automatic tools and all the build recipes are stored under version control in one place
• Debian packages can be updated any way that suits their maintainers, and the build recipes/rules can be stored anywhere (it's the maintainer's job to keep them in version control if they want, then upload them to Debian repositories as source packages)
• Nix (transparently!) caches both build outputs and package sources, which means
◦ if the original source tarballs (e.g., on GitHub or SourceForge) are unavailable, Nix won't even notice that if it can pull them from the ‘binary’ cache
◦ if there is no cache of the build outputs, Nix will automatically fall back to fetching and unpacking the sources from the upstream mirror
• Debian's technical and community relationships to upstream source code are both less robust
◦ Debian requires manual management (creating and uploading) of complete source code archives in their own format[1]
◦ sometimes Debian infrastructure can't even reproduce upstream source code from their own archives[3]
◦ if Debian's source archives are unavailable for a package, there is just no way to build it (since source package archives also contain the build instructions, dependency metadata, etc.)
One way to conceptualize the qualitative differences in reproducibility outlined above is by examining the ways that Nix strengthens Debian's definition of reproducibility[4], which reads:
> A build is reproducible if given the same source code, build environment and build instructions,
For Nix, the build instructions can simply encode all of what Debian calls the ‘relevant attributes of the build environment’:
> Relevant attributes of the build environment would usually include dependencies and their versions, build configuration flags and environment variables as far as they are used by the build system (eg. the locale). It is preferable to reduce this set of attributes.
And similarly, for NixOS, the acquisition of source code is folded into the build instructions and the ‘build environment’ (i.e., caches being available or GitHub not being down). So every Nix package that is reproducible at all is reproducible in a more general way than a reproducible Debian package.
And NixOS/Nix have had to do real work to make their systems reproducible in ways that Debian is not. Unlike much of Debian's work its benefits can't really be shared with distros of a different design— but the converse is sometimes true as well. For example, Debian's work on rooting out non-determinism in package post-install hooks[5] is useless (and unnecessary) for NixOS, Guix, and Distri, since their packages don't have post-install hooks.
There are also lots of little ways that issues Debian has worked on either reflect the relative weakness of this notion of reproducibility (e.g., ‘All relevant information about the build environment should either be defined as part of the development process or recorded during the build process.’[6] is a way of saying ‘the build environment should be reproducible or merely documented’) or overcoming challenges that systems designed with reproducibility in mind from the start simply don't face.
At the same time, the Reproducible Builds website refers to publications[7] by former Nix developers who directly cite the original Nix paper from 2004, whereas Debian's effort didn't begin in earnest until 2013.[8]
Compared to the Nix community, Debian is huge. And they've leveraged their collective expertise and considerable volunteer force to do a ton of work toward reproducible builds which has benefited reproducibility for everyone, including NixOS. Doubtless every remotely attentive member of the Nix community is grateful for that work, which a small community like Nix's could hardly have taken up on its own. But Nix has been attacking reproducibility issues at a different level (reproducing build environments, source code, and whole systems (in terms of behavior, if not bits)) in a meaningful way since long before Debian's reproducible builds effort got going. And some of those efforts have informed the wider reproducible builds effort, just like some of Debian's efforts have not been applicable to every project in the F/OSS community which is interested in reproducible builds.
So: let's praise Debian loudly and often for their work here and be clear that NixOS' reproducibility couldn't be where it is today without that work... but let's also be clear that Nix/NixOS absolutely has blazed some trails in the territory of reproducibility— a terrain that both communities are still mapping out together. :)
—
1: https://michael.stapelberg.ch/posts/2019-03-10-debian-windin...
2: https://wiki.debian.org/ReproducibleInstalls/LiveImages
3: https://www.preining.info/blog/2014/06/debian-pristine-tar-p...
4: https://reproducible-builds.org/docs/definition/
5: https://reproducible-builds.org/docs/system-images/
6: https://reproducible-builds.org/docs/recording/
7: https://reproducible-builds.org/docs/publications/
8: https://wiki.debian.org/ReproducibleBuilds/History#Kick-off
[0] https://www.bootstrappable.org/
[1] https://github.com/fosslinux/live-bootstrap/blob/master/part...
The GNU Mes C experiment mentioned above was also conducted during the 2019 Reproducible Builds summit in Marrakesh.
Critically, it shouldn't be taken as a demonstration that the toolchain is trustworthy unless you trust whoever's picking the compilers! This kind of ruins approaches based on having any particular outside organization certify certain compilers as "trusted".
Your entire effort is potentially brought down by someone making a change in a way you didn't expect and someone goes "huh, that's funny..."
Edit: I think that tone may have come off as snarky, but I meant it as an honest question. If any expert can answer I'd really appreciate it.
Diverse double compilation is useful for run-of-the mill programs, too.
However, the behaviour of programs built by different compilers should be the same. Run-of-the-mill programs could use this as part of a test suite, for example; but diverse double compilation goes a step further:
We build compiler A using several different compilers X, Y, Z; then use those binaries A-built-with-X, A-built-with-Y, A-built-with-Z to compile A. The binaries A-built-with-(A-built-with-X), A-built-with-(A-built-with-Y), A-built-with-(A-built-with-Z) should all be identical. Hence for 'fully countering trusting trust through diverse double-compiling', we must compile compilers https://dwheeler.com/trusting-trust/
Building a project is just a shell script with a couple of defined functions. Quite literally.
I really admire NixOS's philosophy of pushing the boundaries as a distro where everything, including configurations and modifications, can be done in a reproducible manner. They're basically trying to automate the review process down the line, which is absurdly complex as a challenge.
And given stability and desktop integrations improve over time, I really think that Nix has the potential to be the base for easily forkable distributions. Building a live/bootable distro will be so much easier, as everything is just a set of configuration files anyways.
Many people don't realize that, but if you get for example mentioned project from github and I do and we compile it on our machines we get a different file (it'll work the same but it won't be exactly the same).
Say we use the same dependencies, we still will get a different files, because maybe you used slightly different version of the compiler, or maybe those dependencies were compiled with different dependencies or compilers. Maybe the project while building inserts a date, or pulls some file. There are million ways that we would end up with different files.
The goal here is to get bit by bit identical files and it's like a Holy Grail in this area. NixOS just appears to achieved that and all packages that come with the system are now fully reproducible.
Preserving parallel execution, but arriving at deterministic outputs, is an interesting and ongoing challenge. With a rich mathematical structure, too.
I'm really surprised such an attack hasn't happened already. It seems so trivial for a determined attacker to take over an opensource project (plenty of very popular projects have just a single jaded maintainer).
The malicious compiler could inject an extra timed event into the main loop for the time the attack is scheduled to begin, but only if it's >3 hours away, which simply retrieves a URL and executes whatever is received.
Detecting this by chance is highly unlikely - because to find it, someone would have to have their clock set months ahead, be running the software for many days, and be monitoring the network.
That code is probably only a few hundred bytes, so it probably won't be noticed in any disassembly, and is only executed once, so probably won't show up in debugging sessions or cpu profiling.
It just baffles me that this hasn't been done already!
If you count npm packages this happened quite a few times already. People (who don't understand security very well) seems to be migrating to python now.
It's not that reproducible builds provide 0 value it's that they don't truly solve the trust problem as initially stated. They also have non-security value to boot which is often understated compared to the security value IMO.
Most of the world is happy enough with the soft guarantee of: “This is _probably_ your bank’s real website. Unless a nation state is misusing their control over state owned certificate authorities, or GlobalSign or LetsEncrypt or whoever has been p0wned.”
Expecting binary black and white solutions to trust problems isn’t all that useful, in my opinion. Often providing 50% more “value” in trust compared to the status quo is extremely valuable in the bigger picture.
Solving that singular attack vector in the delivery chain does nothing for solving the need to trust the altruism and self interest of maintainers. A good thing™? Absolutely, along with the other non security benefits, but has nothing to do with needing to trust maintainers or be in the niche that reviews source code when automatic updates come along as originally sold.
That same edgewise applies to your bank too. Pinned TLS certs or pre shared keys might help against "BadGuys(tm)", but you're still screwed if your bank decides to keep your money. (s/bank/online crypto wallet/ for real world examples there...)
If you have a fair suspicion that something is up and you discover that when you compile reproduceable-package you get a different output than when you download a prebuilt reproduceable-package, you've now got something to work with.
Your observation that they don't truly solve the trust problem is true. But it's somehow not relevant. It is better to be better off.
Eg suppose you have a software package X, available both as a binary and in source.
With reproducible builds, you can start distributing the binary to your fleet of computers, while at the same time you are kicking off the build process yourself.
If the result of your own build is the same as the binary you got, you can give the command to start using it. (Otherwise, you quarantine the downloaded binary, and ring some alarm bells.)
Similarly, you can re-build some random sample of packages locally, just to double-check, and report.
If most debian users were to do something like that, any tempering with the debian repositories would be quickly detected.
(Having a few malicious users wouldn't hurt this strategy much, they can only insert noise in the system, but not give you false answers that you trust.)
So was most every part of computer hardware and software initially - this is just another milestone in that journey.
I'd imagine that if I were looking at causing world wide chaos, I'd love nothing better than getting into the tool chain in a way that I could later on utilise on a wide spread basis.
At that point I would have achieved my aims and if that means I've burnt a few people along the way, so be it, I'm a bad guy, the damage has been done, the objective met.
What we can't do is throw our hands up and say anyone who compromises the toolchain deep enough is just allowed to win. It will happen at some point if we don't put the right barriers in place.
It's the first step of a long journey, but it is a step we should be taking.
It's true that no single person can audit their entire dependency tree. But many eyes make all bugs shallow.
This is just objectively wrong. I have worked on projects at FAANG where entire teams did not spot critical security issues during review.
You are very unlikely to spot an issue with just one pair of eyes. You need many if you want any hope of catching bugdoors.
For something like C, all bets are off: http://www.underhanded-c.org/ or https://en.wikipedia.org/wiki/Underhanded_C_Contest
> The norm today is auto-updating, pre-built software.
This is a little bit misleading. The actual paper[1] explains that you can't even trust source available code.
[1] https://users.ece.cmu.edu/~ganger/712.fall02/papers/p761-tho...
Say the posgres binary or image is set to encrypt the data on a certain date. Then it asks you to pay X ZEC to a shielded address to get your decryption key. This would work even if the actual database was airgapped.
How the Solarwinds incident, with about every large software vendor being silently compromised for years does not qualify?
Because it does not, people's imagination is as closed as it always was.
So at least for software libraries building from source is definitely viable. Fro end user applications it's another story though, doubt we will ever be at a point where building your own browser from source makes sense...
The upthread hypothetical of a compromised package manager equally applies to a compromised source repo.
_Maybe _ you always check the hashes? _Maybe_ you always get the hashes from a different place to the code? _Maybe_ the hypothetical attacker couldn’t replace both the code you download and the hash you use to check it?
(And as Ken pointed out decades ago, maybe the attacker didn’t fuck with your compiler so you had lost before you even started.)
Only if you define "norm" as what's prevalent in consumer electronics and phones. Certainly, if you go by numbers, it's more common than anything else.
That's not due to choice, though, it's because of the desires of corporations for ever more extensive control of their revenue streams.
>- Build from source. This will always be a deeply niche thing to do. It's slow, inconvenient, and inaccessible except to nerds.
if you trust the compiler :)
You are bringing up red herrings about trusted partners being malicious and untrustworthy.
Do you genuinely believe we should only solve a problem if it leads to a perfect outcome?
So far exploits on FOSS kind of prove the point not everyone is using Gentoo, reading every line of code on their emerged packakges, let alone similar computing models.
Now if we are speaking about driving the whole industry to where security bugs, caused by using languages like C that cannot save us from code reviews unless done by ISO C language lawyers and compiler experts in UB optimizations, are heavily punished like construction companies are for a fallen bridge, then that would be interesting.
How are you measuring the ROI of security efforts inside an OSS distro like debian or nixos? The effort in such orgs is freely given, so nobody knows how much it costs. And how would you calculate the return on attacks that have been prevented? Even if an attack wasn't prevented you don't know how much it cost, and you might not even know if it happened (or if it happened due to a lapse in debian.)
>So far exploits on FOSS kind of prove the point not everyone is using Gentoo, reading every line of code on their emerged packakges, let alone similar computing models.
Reproducible builds is attempting to mitigate a very specific type of attack, not all attacks in general. That is, it focuses on a specific threat model and countering that, nothing else. It's not a cure for cancer either.
>Now if we are speaking about driving the whole industry to where security bugs, caused by using languages like C that cannot save us from code reviews unless done by ISO C language lawyers and compiler experts in UB optimizations, are heavily punished like construction companies are for a fallen bridge, then that would be interesting.
This is just a word salad of red herrings. Different people can work on different stuff at the same time.
That was his Turing Award ;) not Unix as one would assume.
Why? How will this help with the problems you're talking about?
I can't come up with a single benefit to security from reproducible builds. It seems nice for operational reasons and performance reasons though.
It is a means to allow to detect a compromised supply chain. If people rebuilding a distro cannot get the same hash as the distro shipping from the distributor, then likely the distributors infrastructure has been compromised
This doesn't help against the sources being owned, but it helps about build machines being owned.
Accountability for source integrity is in theory provided by the source control system. Accountability for the build machine integrity can be provided by reproducible builds.
To answer your specific questions: The attacker has access to the distro's build servers and is packaging and shipping altered binaries that do not correspond to the sources but instead contain added malware.
Reproducible builds allow third parties to also build binaries from the same sources and once multiple third parties achieve consensus about the build output, it becomes apparent that the distro's build infrastructure could be compromised.
I suppose that's reasonable. Sounds like reproducible builds is a big step towards that, though clearly this requires quite a lot of infrastructure support beyond just that.
Probably not fit for replacing Grandma's budget PC but then again grandma probably isn't worried about the ARM cross compile performance of their machine running NixOS either.
(I am worried about the ARM cross compile performance of my machine running NixOS)
Might be wrong but isn't this part of the premise for Guix/GuixSD?
But it can't do any of the things I bought my computer to do, so it's of limited value to me.
Anyone who has bought an Android phone in the past 5 years knows that's not true.
The problem is unmaintainability, I can’t imagine it’d be easier for medium sized teams where security isn’t a priority, either.
I prefer compiling from source to binary packages. For me it is neither slow, incovenient nor inaccessible.
Only with larger, more complex programs does compiling from source become a PITA.
The "solution" I take is to prefer smaller, less complex programs over larger, more complex ones.
If I cannot compile a program from source relatively quickly and easily, I do not voluntarily choose it as a program to use daily and depend on.
For compiling OS, I use NetBSD so perhaps I am spoiled because it is relatively easy to compile.
That said, I understand the value of reproducible builds and appreciate the work being done on such projects.
I'd love to try building NetBSD, btw, I must try that!
Who remembers Ken Thompson's "Reflections on Trusting Trust"?
The norm today is auto-updating, pre-built software.
This places a ton of trust in the publisher. Even for open-source, well-vetted software, we all collectively cross our fingers and hope that whoever is building these binaries and running the servers that disseminate them, is honest and good at security.
So far this has mostly worked out due to altruism (for open source maintainers) and self interest (companies do not want to attack their own users). But the failure modes are very serious.
I predict that everyone's imagination on this topic will expand once there's a big enough incident in the news. Say some package manager gets compromised, nobody finds out, and 6mo later every computer on earth running `postgres:latest` from docker hub gets ransomwared.
There are only two ways around this:
- Build from source. This will always be a deeply niche thing to do. It's slow, inconvenient, and inaccessible except to nerds.
- Reproducible builds.
Reproducible builds are way more important than is currently widely appreciated.
I'm grateful to the nixos team for being beating a trail thru the jungle here. Retrofitting reproducibility onto a big software project that grew without it, is hard work.