janjongboom

This false sense of reproducability is why I funded https://docs.stablebuild.com/ some years ago. It lets you pin stuff in dockerfiles that are normally unpinnable like OS package repos, docker hub tags and random files on the internet. So you can go back to a project a year from now and actually get the same container back again.

Still gonna break everyone’s CI until they manually update the tag. (And who guarantees that these tags will stay alive after they pull this)

The removal (or moving) of the Bitnami images from Docker Hub is going to break a ton of systems that depend on them. I helped set up https://www.stablebuild.com/ some years ago to counter these types of issues, it provides (among other things) a transparent cache to Docker Hub which automatically caches image tags and makes them immutable - underlying tag might be deleted or modified, but you’ll get the exact same original image back.

Seconding this - absolutely terrific content.

FYI, I've helped set up StableBuild (https://www.stablebuild.com) to help pin stuff in Docker that's normally virtually impossible to pin (e.g. OS package repos, Docker base images, random files from the internet, etc.)

Not never. E.g. all the capital we as founders put in the business before we raised our seed round was converted into Series Seed Preferred shares at the same rights as angels / seed VC. Small portion of total equity but still.

Packages and versions can be deleted from PyPI, which can be a massive pain in the ass for anyone consuming these packages. Can have your whole Python dependency tree pinned => author pulls a package version => builds broken. As part of StableBuild (https://www.stablebuild.com) we create full daily snapshots of the PyPI registry - so figured it would be nice to make an overview of deleted packages/versions and make the wheels available for download.

E.g. jaxlib 0.4.4 was removed a few days back: https://dashboard.stablebuild.com/pypi-deleted-packages/pkg/... => can download the wheels for free w/o registration

I’m an investor in StableBuild.

Yes, fantastic work. Downside is that snapshot.debian.org is extremely slow, times out / errors out regularly - very annoying. See also e.g. https://github.com/spesmilo/electrum/issues/8496 for complaints (but it's pretty apparent once you integrate this in your builds).

> Any source of that claim?

Any tag like ubuntu:20.04 -> this tag gets overwritten every time there's a new release (which is very often)

https://hub.docker.com/r/nvidia/cuda -> these get removed (see e.g. https://stackoverflow.com/questions/73513439/on-what-conditi...)

Yeah, very similar approach. We did this before, see e.g. https://www.stablebuild.com/blog/create-a-historic-ubuntu-pa... - but then figured everyone needs exactly the same packages cached, so why not set up a generic service for that.

Until someone overwrites or deletes the Docker base image (regularly happens), or when you depend on some packages installed through apt - as you'll get the latest version (impossible to pin those).

Yeah, but it's impossible to properly pin w/o running your own mirrors. Anything you install via apt is unpinnable, as old versions get removed when a new version is released; pinning multi-arch Docker base images is impossible because you can only pin on a tag which is not immutable (pinning on hashes is architecture dependent); Docker base images might get deleted (e.g. nvidia-cuda base images); pinning Python dependencies, even with a tool like Poetry is impossible, because people delete packages / versions from PyPI (e.g. jaxlib 0.4.1 this week); GitHub repos get deleted; the list goes on. So you need to mirror every dependency.

Ah, yes, on mobile it shows the wrong pricing table... Copying here while I get it fixed:

Free => Access to all functionality, 1 user, 15GB traffic/month, 1GB of storage for files/URLs. $0

Pro => Unlimited users, 500GB traffic included (overage fees apply), 1TB of storage included. $199/mo

Enterprise => Unlimited users, 2,000GB traffic included (overage fees apply), 3TB of storage included, SAML/SSO. $499/mo

Apologies for editorializing the title a bit :-)

Reading through the comments in https://news.ycombinator.com/item?id=39720007 I saw a common misconception pop up again: Dockerfiles are not deterministic. But they _look_ like they are, and even are for a while: Build a Dockerfile on your local machine; then build it again => most likely exactly the same container. Stuff starts to break down quickly though; so I did this writeup some time ago that should be informative for the wider community.

And that assumes that `foo` and `bar` are not overwritten or deleted in your package repository, and that the git repository remains available.

That's the interesting bit about Dockerfiles. They look _looks_ deterministic, and they even are for a while while you're looking at it as a developer. I've done a detailed writeup of how it's not deterministic in https://docs.stablebuild.com/why-stablebuild

I've gone down the same path. I love deterministic builds, and I think Docker's biggest fault is that to the average developer a Dockerfile _looks_ deterministic - and it even is for a while (build a container twice in a row on the same machine => same output), but then packages get updated in the package manager, base images get updated w/ the same tag, and when you rebuild a month later you get something completely different. Do that times 40 (the number of containers my team manages) and now fixing containers is a significant part of your job.

So in theory Nix would be perfect. But it's not, because it's so different. Get a tool from a vendor => won't work on Nix. Get an error => impossible to quickly find a solution on the web.

Anyway, out of that frustration I've funded https://www.stablebuild.com. Deterministic builds w/ Docker, but with containers built on Ubuntu, Debian or Alpine. Currently consists of an immutable Docker Hub pull-through cache, full daily copies of the Ubuntu/Debian/Alpine package registries, full daily copies of most popular PPAs, daily copies of the PyPI index (we do a lot of ML), and arbitrary immutable file/URL cache.

So far it's been the best of both worlds in my day job: easy to write, easy to debug, wide software compatibility, and we have seen 0 issues due to non-determinism in containers that we moved over to StableBuild in my day job.

Not sure about Earthly as I’ve never used it, but for Nexus and other artifactory solutions around package repos:

1. Don’t need to set up anything. No need to set up an apt repo and push packages or configure a mirror, because StableBuild already caches the complete registry and thus has everything. 2. Don’t need to think about the complete package list when pushing files to artifactory. Have packages cached from 3 months ago, now want to add another one? Oops, not in Nexus, and the current versions in the Ubuntu package registry are not compatible with your cached versions -> need to update the full dependency tree. 3. Integration is trivial. Three lines to your Dockerfile and done. 4. Can retroactively fix things. Knew that this container built 4 weeks ago? Ok use that as a pin date -> fixed.

Then there’s some stuff, like the immutable Docker pullthrough cache and history pypi mirror that I haven’t seen before (but I’d like to learn if others are doing this :-)).

Things Edge Impulse customers have in production: Sleep stage prediction, fall detection for elderly, fire detection in power lines, voice command recognition on headsets, predicting heath exhaustion for first responders, pet feeders that recognize animals, activity trackers for pets, and many more.

I worked on that for a while, see http://janos.io . Interesting idea, but impractical due to the amount of work that needed to be done per device.

Hey I wrote this :-) But back in 2016 or 2017, so some of the practical stuff might be out of date.