Having build and push as part of the same job is frustrating and I view it as a sign that a CI system is built with the expectation of having everything happen post commit by shoveling money into a CI auto-scaler. I know there's `--no-push`, but that's a poor substitute for independent `build`, `tag`, `push` build steps IMO.
Do you have any way of running / debugging locally with GitLab CI plus kaniko? Can you run your build pipeline locally on your workstation against uncommitted code?
IMO I can build a way better local workflow that allows me to run builds _before_ committing with Drone (`drone exec`). I can toggle between a locally bound Docker daemon or a DIND environment that's going to be virtually identical to the DIND environment on a build runner. The `push` step doesn't run locally plus the secrets needed to push are only accessible from an official runner. I can run it on Windows or Linux (and likely Mac) too.
I've been trying to find a self-hosted CI system that's really good at building Docker (or OCI) images and I don't think any exist. They all have short-comings. Having build, tag, push act like an atomic build step is one of the areas where I think most fail. So many claim to enable repeatable builds but none actually do AFAIK. Whoever writes the Dockerfile needs to know a lot about how images are built to have the slightest chance at creating a repeatable build. A great example is having `apt-get update` in a Dockerfile. That command _always_ returns zero, so by itself it makes builds non-repeatable.
Sometimes I have a tough time reconciling the development industry because things just don't make sense to me. I remember people complaining about Gradle start times so much they came up with the Gradle daemon. Now no one bats an eye at CI based build systems where you have to commit your code, wait for a runner to get provisioned, wait for Docker or the OCI runtime to spin up, and wait for your project to actually build on some anemic VM.
People used to complain about seconds because the wait was "too slow" for good local iteration, but now waiting for minutes is a "good" build system. Seriously WTF?
I guess I got on a bit of a rant...
I want this framed or sewn onto a pillow or something.
It's amazing what we can build, it's baffling what we have built.
No, the idea is to build and push images you can test directly afterwards in the same conditions. With cache and such, build times shouldn't be too long
Gitlab CI just runs shell commands, it's pretty trivial to pull the same image its using in the job, and run the same commands locally.
If you have long CI times, that can hinder development productivity and should be improved as much as possible, or a local replica of the CI needs to be created.
In fact, they already provide a utility for doing this: gitlab-runner exec[0].
[0]: https://docs.gitlab.com/runner/commands/#gitlab-runner-exec
Now what happens when two people push code that make changes to the containers at the same time?
- https://github.com/GoogleContainerTools/kaniko/issues/1392
- https://github.com/GoogleContainerTools/kaniko/issues/970
- https://github.com/GoogleContainerTools/kaniko/issues/875
* Switch to a shell runner
* Put the CI dockerfile into your repo
* Provide an entry script for CI that builds the container on-demand (and manages caching/cleanup) and then runs the tests/whatever inside that container
The point here is that docker/podman provide you with everything you need as long as you have full control. By using gitlab's default CI, you relinquish this control.
After using dind for some time we chose to just mount /var/run/docker.sock and keep using the host machine’s docker instance (mostly for the cache), but all in all dind was working fairly well.
To be honest, to say “you shouldn’t be doing that” is missing the point; one should be able to do anything they want. In my opinion, the root cause here is docker’s client/server model, which is fixed by other container runtimes such as podman and rkt (which unfortunately is deprecated). One should be able to just launch containers as if they were just another process.
You started out with asserting "you're too far into Docker", we bring up valid use cases for docker-in-docker, and then you saying "This herd mentality [..] is really fundamentally problematic" is really not adding a lot to the discussion.
For instance the "how do I compose multiple docker containers" is trivial when you can just execute a script that runs docker or podman. If you really want, you can use docker-compose.
Userland docker builds
I'm not sure why it doesn't do this by default. Performance I guess.
We're using podman in containers inside gitlab ci.
We're _also_ running tests of containers inside containers in containers using gitlab-ci.
The main workaround we've applied is using crun as the runtime rather than runc.
In my experience, that is not true at all. Docker-in-docker allows me to deliver smaller images that can fit into a CI flow as language plugins instead of shipping a beastly 5G docker image with every possible language runtime I need to support for my CI tool.
Docker-in-docker is a workaround to make docker work in CI.
Basically a security nightmare and bad design that podman doesn't have.
Docker in Docker in CI is like a lock on a door. It keeps honest people from being naughty, and is fairly efficient about it.
I don't think the question is "should I run CI in docker in docker," it's whose CI should I run in docker in docker. Me in my coworkers can share docker images. Customers or freeloaders cannot. So if that's in your problem domain, then you're right, it's a bad idea. But it isn't for most people.
This works great if you own or rent the hardware, but most cloud providers don't allow nested virtualization.
Although there are tools to convert docker images to vm images. I expect if I were running community CI infrastructure, getting really familiar with those would be high on my priority list.
The huge issue with that is security which is why it's only really practical for a single user or a small group of trusted users. A secondary issue is that (I think) builds can't run simultaneously because they can trample each other when tagging images (since all images are on the runner's daemon).
If I had to build a Docker focused CI system I'd think about using Weave Ignite (AWS Firecracker) to spin up VMs for runners with the Docker socket bound like described above. That way you get all the convenience of binding the Docker socket, but the isolation of a VM that gets thrown away after the build step (or pipeline) finishes. That idea also fits well with local running / debugging IMO because you can bind to the Docker socket on your development workstation (assuming you're not running a large build of parallel tasks which might be an unrealistic assumption).
You could separate those into two builds, but the reason they are together is so people think about deployment, and in case any structural changes to the code need to coincide with deployment changes. For instance, breaking changes in APIs. I need a new version of tool/library and I need to change how I call it.
I don't think kubernetes is a solution for context of building an image (a rootfs tree into a .tar.gz file).
Unless you are using kaniko which extends the kubernetes api to add the capability of creating images, but that is handled by kaniko itself via the same api.
my beastly 12GB image that even includes Matlab wants a word with you
Perhaps in the next 10 years we will be rediscovering packages. :P
If you are in the business of charging complex prices per bits over the network, then docker seems to be quite a good investment and making it as popular as possible is a good strategy to print money. /s
True, that.
To be fair, at least it allows me to avoid lots of the brokenness of Python packaging.
tl;dr pip silently breaks my environments, mostly connected to upgrading numpy and other scientific/data science libraries.
I use docker for most of my clients' work but for in house stuff I just use nix.
That has not been the case for a good while now... Docker has been running directly on a hypervisor on the Mac.
In Linux iirc, a hypervisor can share such resources with the host system (since they are both Linuxes).
I encourage you to not take the standard workflows as a given and really think about what you need and I bet you either end up with a use case that can be covered by rootless podman or something that requires real VMs anyways.