Consumers often mistake hasn’t changed for a commitment to never change: any sufficiently large product will be littered with these kind of implicit commitments made by the product to consumers that nobody has visibility into. You’re unfortunate that we were all relying on this commitment you’ve never made, but the quick reversion is the best we can hope for. People will theorise how this could have been avoided but c’est la vie — easy mistake that you’ve responded well to.

At this point they'll be stuck on old git for all of eternity unless they just roll their own archive/compress step out of band so the old hashes still work. Yikes.

They could also brownout the implied contract over a longer timespan.

They could use the old behavior for archives in which all the inputs predate the changeover.

New git has a flag for keeping the old behavior, so it's not as bad.

Hyrum's Law:

With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody.

Incidentally, I'm still waiting for GitHub to implement Spacebar Heating: https://m.xkcd.com/1172/

FWIW according to https://github.com/bazel-contrib/SIG-rules-authors/issues/11... a commitment was made, although in an exchange in some support ticket, and not in documentation.

> At the time we didn't foresee the impact.

Are you new to open source software? Because the only way to NOT foresee the impact is to be completely ignorant of how source-based package systems work.

Ironic that "open source packaging systems" rely on proprietary Microsoft hosting and distribution to function.

I think you meant _poorly implemented_ open source packaging systems.

* * *

Or get the source code using Git, which actually (by design) guarantees that its checksums are stable.

Downloading a source tarball is significantly cheaper on both sides than git. A source tarball is 100% served from CDNs, whereas I don't believe the same isn't quite true for git (even over https).

That's a good point.

It's way more resource-intensive and much slower, which is why it's not preferred in Nixpkgs, for example.

But it's also vulnerable to the same problem in that your package Manager's build system is still dependent on GitHub. It will take more to screw you up, but a whole GitHub outage, for example, will definitely still hurt.

It's not depending on github-specific functionality tho. You can just redirect it to another mirror of git repo of the project

Of the 11,656 packages in OpenBSD’s package repos, 2,984 are built from source originally hosted on GitHub or Sourceforge. That’s a full 25%.

Moralize all you want about where these upstreams should host their software, but why claim that the downstream package manager is “poorly implemented” to fetch source code from those hosts? Your complaint was not technical—you imply the proprietariness of Microsoft servers is the problem (although open source servers like GitLab also have the problem of unstable checksums)—but HTTPS is HTTPS.

I'm fairly certain that Homebrew wouldn't function without GitHub. The index (and Cargo's index, and probably others) is hard-coded to be hosted on GitHub.

Exactly this.

Tbh, i would expect some developer working on this feature to have an ‘a-ha’ moment: ‘I’m a homebrew user … hey wait a minute’

> Moralize all you want about where these upstreams should host their software, but why claim that the downstream package manager is “poorly implemented” to fetch source code from those hosts?

Because it should validate checksums of content of the tarball, instead of just the outside blob.

Then:

* you don't care about compression method or implementation * you don't care about archive method or implementation * your system works just as well for "download a tarball" as for "shallow copy the remote repo

I am not sure how you could not rely on GitHub when packaging code that is hosted on GitHub.

My personal Gentoo ebuilds for example contain a URI variable that points to the GitHub auto-generated archives for projects that use GitHub for hosting. What am I supposed to do in this case?

The only option here is to setup a mirror and have a backup of the data. Packages that are in the official Gentoo repository do get mirrored, but overlays do not, and most people probably don't have the ability, time or money to setup their own mirroring service.

I agree that relying on GitHub sucks to be clear, but I don't think we can blame package managers for fetching code from it when people are hosting their projects there!

Software Heritage backs up GitHub, they aren't really designed to be a mirror though.

https://www.softwareheritage.org/

Some GitHub repos also end up on archive.org, but not systematically.

Most source-based pkg managers build from release tarballs, or even preprocessed tarballs after doing autoconf

Sadly there has been a sharp uptick in software that provides no release tarballs anymore. With the rise of GitHub many upstreams choose to make a tag and let people download the autogenerated tarballs, despite the fact that they won’t contain preprocessed autoconf or (more importantly) any Git submodules.

The situation is deteriorating further as some projects make no releases at all, assuming users will add the project’s own package mirror to the user’s trusted package repositories, or use Docker.

> many upstreams choose to make a tag and let people download the autogenerated tarballs

Which is fine as long as you rely on the hash of the tag rather than the hash of the tarball.

> despite the fact that they won’t contain preprocessed autoconf

This is a feature; run `autoreconf -vfi` at build time, so that you don't depend on the maintainer's idiosyncratic autotools setup and local macros, and so that you can reliably regenerate it all if you want to change configure.ac or Makefile.am.

> This is a feature; run `autoreconf -vfi` at build time, so that you don't depend on the maintainer's idiosyncratic autotools setup and local macros, and so that you can reliably regenerate it all if you want to change configure.ac or Makefile.am.

On a package bulk build machine, that’s a lot (like, a lot) of wasted CPU cycles multiplied by the thousands of packages that use autoconf. For the majority of packages that don’t patch configure.ac or Makefile.am, it’s nicer to use a preprocessed tarball and check that you can reproduce the same autoconf output when adding the package to the package manager, because then it only happens once.

There are a lot of things that waste cycles on a build machine, but they're worth the reproduciblity. I think that problem would be better solved via caching, ideally.

I would hazard a guess that there are far fewer people these days who download a tarball and `./configure` `make` `make install` than there are distros (who often need to patch) and developers (who will be working from git anyway).

Having autocruft in the default tarball is a design flaw in autotools, you should never ship prebuilt files in git nor in source tarballs. I think `make distcheck` should by default put the autotools files into a separate foo-1.2.3.4-2023-01-31-autocruft.tar alongside the real source tarball generated by git-archive.

The distros always run `autoreconf` these days, so that they can verify that they can still build the build system from the source configure.ac/Makefile.am files.