News for hackers. Stuff that matters.
Re:Disorderfs: FUSE-based filesystem that introduces non-det... by infogulch on Friday January 29, 2021 @05:53PM (#25963876)
If you are confident that certain build steps are deterministic, can you enable Dockerfile-like caching for intermediate steps? The way docker does it is take a hash of the "input" filesystem and the command, and see if there's an associated "result" filesystem, and if there is then just jump to evaluating the next command with the previous result as input.
Re:Disorderfs: FUSE-based filesystem that introduces non-det... by ithkuil on Friday January 29, 2021 @06:08PM (#25964027)
That's precisely what bazel does (and remote build execution systems like buildbarn allow this cache to be effectively shared among users)
Re:Disorderfs: FUSE-based filesystem that introduces non-det... by infogulch on Saturday January 30, 2021 @02:24PM (#25972077)
Ah, cool! IIRC Bazel predates docker, so perhaps it's more correct to say that dockerfiles use a Bazel-like command caching strategy.
Re:Disorderfs: FUSE-based filesystem that introduces non-det... by ithkuil on Saturday January 30, 2021 @04:32PM (#25973219)
Yep, moreover bazel has a fine grained dependency graph and thus can invalidate only a minimal part of the action graph when some inputs change.

Docker instead has to re-build all subsequent layers when the input of even just one layer changes.

The two tools sit on a different point in the spectrum of simplicity of use though. Maintain build files for bazel (and dealing with other constraints of hermetic execution) is hard time-consuming and it's hard to convince many teams it's worth the effort.

Docker apparently struck a sweet spot in that it's easy to explain how to craft linear build steps and it does a half-decent job in actually caching stuff. Sometimes it doesn't work well depending on your workflow but people then have the incentive to read about how to improve the "cacheability" of their dockerfiles (multistage, reorder, dockerignore, ...). As many things in our craft, the human aspect trumps over technical brilliance.