I never got around to submitting my blog to HN. But if people are curious about some technical details of the underlying problems of reproducing Arch Linux packages I wrote something a few months ago.

https://linderud.dev/blog/reproducible-arch-linux-packages/

EDIT: I did self-discover I did submit the article when I wrote it. Just forgot. Oh well :)

This is fantastic to see. Reproducible builds adds yet another layer of trust on top of open source software. The wiki page is also classic Arch Wiki-style, with all the detail one would expect. Every OSS project should strive to be this helpful.

I care more about if maintainers actually audit the contents of packages rather than if their builds are reproducible (though the latter still matters)!

Not just is there obvious malware, but also are there obvious vulnerabilities, is the person that wrote it of good nature / located in a country where they’re safe from nation state pressure, is there a lot of history behind the app.

Obviously this is too much work for any individual and requires a chain of trust. I believe fedora and Ubuntu at the very least audit to some extent but I’ve never seen any doco.

Just curious, are you saying there is obvious malware within the Arch official repositories? Are you referring to instead the user repositories (AUR)?

I know Arch has: https://wiki.archlinux.org/index.php/Arch_Security_Team

but I'd be really interested to hear that the official packages have obvious malware.

I think they intended the phrase as a question; they're proposing that a maintainer should not just ask "is there obvious malware in this package?", but also several other questions (which they list afterwards).

This is the correct interpret

Isn't this chain of trust essentially what crev [0] tries to do? FWIW Rust implements cargo-crev [1], but I suppose you could extend this to AUR packages with a bit of work.

[0] https://github.com/crev-dev/crev/

[1] https://github.com/crev-dev/cargo-crev

Debian have done a huge amount of work in this area - https://wiki.debian.org/ReproducibleBuilds

Makes sense, given Chris Lamb, former Debian Project Leader, is heavily involved. https://reproducible-builds.org/who/

I believe the whole reproducible-builds.org project started at Debian [0], the Core Team is still all made of Debian Developers, more information at: https://reproducible-builds.org/

Their documentation has some really good pointers at common issues and how to solve them: https://reproducible-builds.org/docs/

And the diffoscope tool is very useful for debugging.

[0] I believe other projects had similar things going on but the Debian one became the main to serve as an umbrella.

Some more reproducible builds history (it has been around since at least the 1990s):

https://wiki.debian.org/ReproducibleBuilds/History

Based on my reading of this Arch wiki page[0], it looks like this doesn't impact packages in the AUR. Does anyone who is more familiar with this know if that is true? It appears that it does work for community packages though (see the bottom of this page[1]).

[0] https://wiki.archlinux.org/index.php/Rebuilderd

[1] https://wiki.archlinux.org/index.php/Rebuilderd#Syncing_pack...

>Based on my reading of this Arch wiki page[0], it looks like this doesn't impact packages in the AUR.

Yes. AUR is excluded from reproducible builds because it only provides build scripts and not actual packages. There is nothing to reproduce. One surely could make a repo and compare results, but it would frankly be a bit futile considering the general package quality there.

AUR packages are source only so there's no binary package that is distributed to be verified.

that's only mostly true.

take for example the aur package for MS fonts; https://aur.archlinux.org/packages/ttf-ms-fonts/

all it does is download a ton of microsoft corefont executables to unpack.

another example would be the proprietary driver packages, like the nvidia ones.

This is a question of the artifacts distributed as part of Arch Linux, the tar.xz packages. The idea is to be able to reproducibly build a package from source (where "source", yes, may include binary assets and blobs), and check that the copy you built matches what Arch distributes, byte for byte. This way, you know the packager hasn't introduced unexpected changes.

Since end users build AUR packages themselves, there is nothing distributed, and nothing to verify.

You're describing what those AUR builds do not what they are.

Reproducible builds are for artifacts that are... built. Plenty of AUR recipes download blobs and explode them, but the resulting xz package isn't actually hosted in the AUR.

A tool like rebuilderd would only be relevant to the AUR in the case that somebody is publishing generated AUR packages to a repo for others to install without building them directly, and then you'd like to verify those.

I think of the AUR packages as more like ebuilds with fewer safety catches 8)

proprietary drivers from nvidia are provided by the Arch mainline (core) though?

right you are, i've been on beta for years and had forgotten.

point still stands, aur does distribute executables once in awhile.

> aur does distribute executables once in awhile.

The AUR doesn't distribute those executables; it distributes the instructions for the user (or some software working on behalf of the user) to go fetch the executables themselves. This is an important distinction, because the official Arch repositories _do_ distribute executables, directly to the user.

Interesting to see https://github.com/kpcyrd/rebuilderd in Rust. Is this the first Arch project in Rust?

I guess,.. but this FOSDEM we hacked on more Rust based tooling :-)

https://github.com/archlinux/rebuilder https://github.com/archlinux/signstar

This is very good news. I was wondering recently just how to do this, as I wanted to build everything with some different compiler configuration.

The wiki says:

> a large number of builds are not reproducible yet

Yes. But a staggering amount of packages in the main [core] repository is still fully reproducible by independent parties if tools are run.

https://wiki.archlinux.org/index.php/DeveloperWiki:Reproduci...

It's still very much a work in progress.