I wish that docker has the ability to merge multiple parent layers like git, then you can build the gigantic image by just updating single layer.
The only hack the docker can do is multistage-build, however that won't work reliably in some cases such as resolving conflicts.
There is actually the --squash command that you can use during builds, to compress all of the layers: https://docs.docker.com/engine/reference/commandline/build/#...
For example:
$ docker build --squash -t my-image .
E.g.:
- layer N: whatever the base image needs
- layer N+1: whatever system packages your container needs
- layer N+2: whatever dependencies your application needs
- layer N+3: your application, after it has been built
Also, the above order also helps immensely with Docker build caching. No changes in your pom.xml or whatever file you use for keeping track of dependencies? The cached layers on your CI server can be used, no need to install everything again. No additional packages need to be installed? Cache. That way, you can just rebuild the application and push the new layer to your registry of choice, keeping all of the others present.
Using that sort of instruction ordering makes for faster builds, less network traffic and ergo, faster redeploys.
I even scheduled weekly base image builds and daily builds to have the dependencies ready (though that can largely be done away with by using something like Nexus as a proxy/mirror/cache for the actual dependencies too). It's pretty good.
Edit: actually, i think that i'm reading the parent comment wrong, maybe they just want to update a layer in the middle? I'm not sure. That would be nice too, to be honest, though.
The image contains dependencies needed for 50+ languages. This means repls by default are packed with lots of commonly used tools. However, the image is massive, takes a long time to build, and is difficult to deploy.
Unfortunately, slimming the image down is not really an option: people rely on all the tools we provide out of the box.
May I ask why you didn't use something like Ansible to build such a complex image? With appropriate package version pinning (which it's the real crux here) it should work well enough to get a reproducible build.
I understand it would already have been something different from a pure Dockerfile so it's not that fair to compare buuut...
They did; it's called Nix, and they wrote a blog post about it ;)
We've been able to utilize Nix to address both of those issues, and others who may be in a similar scenario might also find Nix to be valuable.
Of course Nix comes with its own set of opinions and complexities but it has been a worthwhile trade-off for us.