News for hackers. Stuff that matters.
Re:A tale of several distros joining forces for a common goa... by jmclnx on Saturday February 08, 2025 @09:37AM (#42983202)
It is very cool to see distros working together for a common goal.

But I still do not understand the point of "reproducible builds". I know what they are, but to me the amount of work involved outweighs the benefit.

I even heard NetBSD is also working on "reproducible builds". So maybe I am missing something :)

Re:A tale of several distros joining forces for a common goa... by noirscape on Saturday February 08, 2025 @12:01PM (#42984299)
Practically speaking, the idea with a reproducible build is that you can take the source files they used, run their instructions the same way they did and get hash for hash[0] the specific resulting executable.

The main benefit is that you can trust that the resulting binary file being served matches the source code that it's build from. This mostly matters for distros in that they build from a source package repository, but anyone running a mirror could hypothetically replace the package with another (potentially malicious) package, leading users to install malicious tooling. It mainly matters for distros because pretty much every distro out there runs on third party mirrors (often ran by universities, but also just people who want to help) rather than on direct upstream; packages get uploaded to a main server, then mirrors copy from that main server (to reduce network traffic load on the main server). Right now, mirror trust is mostly "we assume you're not gonna be evil, until we get complaints". If the build is reproducible, the software can inherently confirm that the file they're getting is trustworthy, making "getting complaints" much easier to confirm.

It can also speed up the overall building process; if the package source code hasn't changed, you can also always assume that the resulting binary hasn't changed (meaning you can use hashes instead of relying on mtime like make does). Docker build cache works in a somewhat similar way (although docker isn't inherently deterministic).

Devwise, you can also reconstruct a build much easier if it's reproducible; ie. if you've accidentally thrown away the .elf file for debugging, if your build is deterministic, you can just rerun the build and get the same .elf file again.

[0]: While not a problem for Linux distros, in cases where you need a secret to sign an application, reproducible typically means "identical except for the signature" instead. F-Droid uses this for example to figure out if they should use buildserver stuff or the original APKs: https://f-droid.org/docs/Reproducible_Builds/

Re:A tale of several distros joining forces for a common goa... by yellow_lead on Saturday February 08, 2025 @12:35PM (#42984540)
> a mirror could hypothetically replace the package with another (potentially malicious) package, leading users to install malicious tooling.

It was my assumption that a mirror is required to host a build that has a hash conforming to the original. Is that not the case?

Re:A tale of several distros joining forces for a common goa... by jerf on Saturday February 08, 2025 @12:56PM (#42984698)
Yes, the real attack isn't that mirrors change the files, the real attack is that just because a distro packages Binary X and Source X, it is difficult without reproducible builds to prove that Source X actually did produce Binary X. It could have been compiled with a trojan in it between the source and binary.
Re:A tale of several distros joining forces for a common goa... by gruez on Saturday February 08, 2025 @12:39PM (#42984566)
More specifically the packages are signed by the distro and automatically checked, so a mirror can't go rogue even if it wanted to.
Re:A tale of several distros joining forces for a common goa... by SkiFire13 on Saturday February 08, 2025 @01:44PM (#42985126)
> meaning you can use hashes instead of relying on mtime like make does

Note that mtime still has the advantage of being faster than hashing.

Re:A tale of several distros joining forces for a common goa... by patmorgan23 on Sunday February 09, 2025 @03:45PM (#42993778)
>but anyone running a mirror could hypothetically replace the package with another (potentially malicious) package, leading users to install malicious tooling.

I thought all packages were cryptographically signed, and that the package manager would compare the hashes of artifacts downloaded from mirrors to the hashes listed in the package index (which is also signed). This is not an attack that needs reproducible builds to mitigate.

Re:A tale of several distros joining forces for a common goa... by david-gpu on Saturday February 08, 2025 @09:49AM (#42983281)
It's a safety measure. Reproducible builds ensure identical binaries are produced from the same source. They help detect e.g. hidden backdoors.
Re:A tale of several distros joining forces for a common goa... by champtar on Saturday February 08, 2025 @11:19AM (#42983987)
With reproducible build you know that what you test on your dev laptop is the same as what will go out from your CI, and if hash mismatch you can chase why. For a concrete exemple, Mellanox driver configure script will auto detect if it's running under docker and change a compile flags, so if you build in a container using podman you get a different result.
Re:A tale of several distros joining forces for a common goa... by solarkraft on Saturday February 08, 2025 @11:24AM (#42984033)
> but to me the amount of work involved outweighs the benefit

I don’t know whether I’d spend this much work on such an abstract goal, but what reproducibility changes really is quite amazing. It vastly increases trust in published binaries and obviates the need for signing and the security benefit of compiling software yourself.

Re:A tale of several distros joining forces for a common goa... by gruez on Saturday February 08, 2025 @12:48PM (#42984632)
>obviates the need for signing and the security benefit of compiling software yourself

Not really. Most people still would rely on signatures because they can't be expected to compile everything from scratch just to verify their download is authentic. Moreover even though reproducible builds make verification easier, it still requires someone to sound the alarm. For less popular packages there might be nobody checking any particular build is backdoored, because most people see "reproducible builds" and they assume Somebody Else is doing the reproduction.

Re:A tale of several distros joining forces for a common goa... by mjl- on Saturday February 08, 2025 @01:25PM (#42984967)
for transparency of reproducible builds of go applications, i made https://beta.gobuilds.org/. it compiles any publicly available go application on-demand, with a toolchain version of your choice (latest stable by default), for a platform of your choice. all (pure) go applications are reproducible by default, including when cross-compiled, and go toolchains run nothing provided by the go module (awesome properties!). the source code is verified through the go sum database (a transparency log containing go modules). the hash of the resulting binary is added to gobuild's own transparency log. so it can be publicly verified. the gobuilds service builds the binary itself, and has another instance (on a different platform & config) build the binary too, to ensure the binary is really reproducible (i'ld like other instances that i don't run myself as secondaries too). i no longer publish binaries for my applications (that i write in go). i just point to the "latest"-build link for the go module at gobuilds. also makes it easy for users (including myself) to get new builds for new go toolchains (which may include fixes to the (relatively large, and often used) standard library).

you still may not trust the public gobuilds instance. my hope is that people (eg software projects themselves, or distros, or other kinds of communities) will run & use their own gobuild instances and verify their builds against the public gobuilds service. win-win: gives them assurance their builds are really reproducible, and builds trust in the public gobuilds (keeping it honest, if someone sees a hash mismatch, they will speak up).

i usually don't get much enthusiasm for it though. (:

Re:A tale of several distros joining forces for a common goa... by 3s on Saturday February 08, 2025 @10:44AM (#42983667)
A really important application of reproducible builds is running code inside Secure Enclaves that has been committed to on a public transparency log. A client can connect to a remote secure enclave that can then prove to the client that it’s running the commit code via a process known as remote attestation. It’s pretty cool stuff. However it’s only possible if the build inside the enclave is reproducible (deterministic) and always identical to the build on the transparency log
Re:A tale of several distros joining forces for a common goa... by NegativeLatency on Saturday February 08, 2025 @11:13AM (#42983924)
Would make stuff like this harder to pull off: https://en.wikipedia.org/wiki/XZ_Utils_backdoor
Re:A tale of several distros joining forces for a common goa... by nindalf on Saturday February 08, 2025 @11:54AM (#42984239)
Are you sure?

If I'm understanding correctly, the malicious code was introduced as part of the test code, so no matter who compiled it, they'd get a binary with the same (malicious) functionality. Heck, it might even have been reproducibly malicious.

The real crazy part was that it was modifying the functionality of sshd at runtime, allowing the attacker to log into any system.

Reproducibility of either sshd or xz wouldn't have stopped this attack.

That's my reading of https://research.swtch.com/xz-script.

Re:A tale of several distros joining forces for a common goa... by solarkraft on Saturday February 08, 2025 @12:47PM (#42984626)
I tend to agree. The exploit was (by detour) committed to the source.
Re:A tale of several distros joining forces for a common goa... by ramses0 on Saturday February 08, 2025 @02:08PM (#42985332)
Technically, interestingly, if `bazel` were used as the build tool, it would avoid the straightforward ability to cross-contaminate the build executable with the test code...

Yeah, `bazel run test:...` would have access to the test files, but `bazel build xz:executable` would not (by default) be able to pull in extra shenanigans from the test files (and I think there's generally linting and formatting rules required by default with `BUILD.bazel` files, reducing another sneak-vectors)

Re:A tale of several distros joining forces for a common goa... by kpcyrd on Saturday February 08, 2025 @01:47PM (#42985151)
It unfortunately doesn't help in cases like this. Reproducible Builds gives you a trusted path from source to binary, but it doesn't help with backdoors in the source code/build instructions.

For that we'd need some sort of source code reviewing effort like https://github.com/crev-dev/cargo-crev implements. I've started whatsrc.org to keep track of the source code inputs we're putting into our computers (that would benefit from reviews), but the conclusion is also somewhat "it's too much".

Re:A tale of several distros joining forces for a common goa... by ssivark on Saturday February 08, 2025 @10:21AM (#42983499)
What makes you so confident that the benefit is less than the effort?

Given the increasing likelihood of supply chain attacks, isn’t this a very prudent precaution?

Re:A tale of several distros joining forces for a common goa... by samsartor on Saturday February 08, 2025 @09:49AM (#42983283)
The video gets into that. The main purpose is to verify that the binary you're running came from the actual source code.
Re:A tale of several distros joining forces for a common goa... by jcranmer on Saturday February 08, 2025 @01:16PM (#42984895)
The main benefit you'll hear touted is something along the lines of being able to get an attestation that the resulting artifact was built following the steps claimed to build it. I think that's a somewhat overstated benefit, though, as it's not clear to me that this is an avenue of attack used in practice, given the frequency with which software already has vulnerabilities usable for exploits, or the ease with which one can insert a backdoor into the source code (e.g., the xz backdoor).

I think the actual main utility is that the process has done a very good job of rooting out several causes of unintentional nondeterminism in the build process. I say unintentional because the two main causes of unreproducibility, by several orders of magnitude, are timestamps being embedded everywhere and absolute paths being embedded everywhere, and those are rather expected. But some of the unreproducibility comes from things like accidental reliance on inodes in file paths (i.e., doing "for file in listdir()" without sorting the results of listdir) or the compiler itself accidentally sorting based on pointer address (which is unreproducible on ASLR systems).

Re:A tale of several distros joining forces for a common goa... by uecker on Saturday February 08, 2025 @03:26PM (#42985882)
The xz backdoor was news because they went to a lot of effort to try to hide a backdoor in the source (actually a binary file in the source) and still failed. In contrast, without reproducible builds it is trivial for a maintainer with upload rights (or somebody who managed to get the credentials from a maintainer) to insert a backdoor into a binary. And it is then virtually impossible to detect.
Re:A tale of several distros joining forces for a common goa... by rcxdude on Saturday February 08, 2025 @07:13PM (#42987343)
Well, the xz backdoor was detected through the behaviour of the resulting binaries, not through observation of the source code tampering, so I don't think it's a great example.
Re:A tale of several distros joining forces for a common goa... by uecker on Saturday February 08, 2025 @08:15PM (#42987627)
Fair point. It was pure luck that it was discovered. What I was referring to mainly was how much effort it took to get the backdoor installed in the first place, compared to simple uploading a compromised binary somewhere.
Re:A tale of several distros joining forces for a common goa... by drowsspa on Saturday February 08, 2025 @07:59PM (#42987569)
Yeah, I would say the x backdoor succeeded summa cum laude
Re:A tale of several distros joining forces for a common goa... by zelphirkalt on Saturday February 08, 2025 @04:39PM (#42986423)
It is also about reliably being able to built things a month from now, in a year, in 5 years, etc.