News for hackers. Stuff that matters.
Re:Rails on Docker · Fly by vidarh on Thursday January 26, 2023 @02:30PM (#34537428)
How so? I just tested a build, and it used the cache for every layer including the builder layers.
Re:Rails on Docker · Fly by orf on Thursday January 26, 2023 @02:33PM (#34537480)
You did this on the same machine, right? In a CI setting with no shared cache you need to rely on an OCI cache. The last build image is cached with the inline cache, but prior images are not
Re:Rails on Docker · Fly by vidarh on Thursday January 26, 2023 @02:53PM (#34537782)
Ah, sorry I misunderstood you. Yes, I don't tend to care about whether or not the steps are cached in my CI setups as most of the Docker containers I work on build fast enough that it doesn't really matter to me, but that will of course matter for some.
Re:Rails on Docker · Fly by korijn on Thursday January 26, 2023 @03:03PM (#34537929)
You can build the first stage separately as a first step using `--target` and store the cache that way. No problem.
Re:Rails on Docker · Fly by orf on Thursday January 26, 2023 @06:21PM (#34540273)
How would you do this in a generic, reusable way company-wide for any Dockerfile? Given that you don't know the targets beforehand, the names, or even the number of stages.

It is of course possible to do for a single project with a bit of effort: build each stage with a remote OCI cache source, push the cash there after. But... that sucks.

What you want is the `max` cache type in buildkit[1]. Except... not much supports that yet. The native S3 cache would also be good once it stabalizes.

1. https://github.com/moby/buildkit#export-cache

Re:Rails on Docker · Fly by korijn on Friday January 27, 2023 @02:57AM (#34543708)
Standardize the stages and names. We use `dev` and `latest`.

It worked wonders for us, on a cache hit the build time is reduced from 10 to 1,5 minutes.

Re:Rails on Docker · Fly by claytonjy on Thursday January 26, 2023 @04:38PM (#34539192)
I never got around to implementing it but I wonder how this plays with cross-runner caches in e.g. Gitlab, where the cache goes to S3; there's a cost to pulling the cache, so it'll never be as fast as same-machine, but should be way faster for most builds, right?
Re:Rails on Docker · Fly by orf on Thursday January 26, 2023 @06:22PM (#34540285)
If you're running the CI inside AWS, and assuming the code isn't doing anything stupid, it will be fast enough for nobody to notice.
Re:Rails on Docker · Fly by maaarghk on Thursday January 26, 2023 @07:54PM (#34541113)
the cache is small but if you have a `docker buildx build --cache-from --push` type command it will always pull the image at the end and try to push it again (although it'll get layer already exists responses), for ~250mb images on gitlab I find this do-nothing job takes about 2.5 mins in total (vs a 10 min build if the entire cache were to be invalidated by a new base image version). I'd very much like it if I could say "if the entire build was cached don't bother pulling it at the end", maybe buildkit is the tool for that job
Re:Rails on Docker · Fly by claytonjy on Thursday January 26, 2023 @11:30PM (#34542490)
I mostly love Docker and container-based CI but wow what a great reminder that even common-seeming workflows still have plenty of sharp edges!