"BUILDX_EXPERIMENTAL=1 docker buildx build <args> --invoke /bin/sh"

There's actually a whole debugging client available.

This is great! I’ve been waiting a long time for this and it seems like more debug features are in the pipeline too: https://github.com/docker/buildx/issues/1104

Docs: https://docs.docker.com/engine/reference/commandline/buildx_...

and its "oh, you want multi-arch, do you?" friend. While prosecuting this <https://gitlab.com/gitlab-org/gitlab/-/issues/339567> I learned that they bundle binfmt into the buildkit docker image <https://github.com/moby/buildkit/blob/v0.12.5/Dockerfile#L11...> which itself is compiled against a static copy of qemu <https://github.com/tonistiigi/binfmt/blob/deploy/v6.2.0-26/D...> with various /proc/sys/fs/binfmt_misc wizardry <https://github.com/tonistiigi/binfmt/blob/deploy/v6.2.0-26/c...>

We ended up switching to using multiarch/qemu-user-static:7.2.0-1 <https://hub.docker.com/layers/multiarch/qemu-user-static/7.2...> which also mutates binfmt_misc in buildx's context in order to exec the static copy of qemu in it but I find their shell script much more legible about what's going on: <https://github.com/multiarch/qemu-user-static/blob/v7.2.0-1/...> not to mention the fact that it more clearly represents the recent qemu version versus secreting it away inside the moby/buildkit:heheh-good-luck-friend

They have an entire GH label for that: https://github.com/docker/buildx/labels/area%2Fqemu and my journey into the buildx sewers started from https://github.com/docker/buildx/issues/1170

I see you're trying to do multi-arch builds. At reasonable (on bare metal) speeds you say?

Delete the entire docker stack. It's utter shite. Then install Podman and Buildah, and rejoice! For all the evil buildx magic can now be done explicitly.

This solution was born when trying to do native builds on CircleCI - where you do not have a buildx cluster with multiple architectures. You have to build the archs as independent build steps, then pull them together afterwards. This is something that Docker never considered, and Docker only functions on the happy path. Buildah has multiarch manipulation commands: https://danmanners.com/posts/2022-01-buildah-multi-arch/ (not a plug, first blog on search)

We migrated to GHA, but still use separate builders to avoid the Docker trite. We also ditched DCT (Docker signing) for Cosign because - again - DCT is home-grown garbage.

If you build your dockerfile with earthly, we have a -i flag that will pop you into a shell on a failure, and we are using buildkit, so it's totally possible.

Well said, these are the reasons I use podman/buildah now.

Was it a drop in replacement or did you need to give things a jiggle to get it going?

Similar to sibling comment, mostly just works out of the box. Things I've noticed that are different:

- There's no docker.sock API socket by default, which makes sense because Podman doesn't have Docker's daemon-based architecture. You can run one with `podman system service [...]` if you have non-CLI clients expecting to connect to some $DOCKER_HOST

- Mounts/volumes behave subtly differently IIRC, I had a bunch of scripts that didn't work with podman-as-docker out of the box, I think the issue was that Docker will create some directories automatically that Podman won't?

Aside from that, rootless Podman is great and I've had far fewer issues with it than I did for rootless Docker, only real caveat is that -p <some port> doesn't actually connect to the host, it's still namespaced. You have to use --net=host for actual host networking.

Only reason I have used buildx is the easy multi-arch for the same tag (big manifest). How to do this with Podman/buildah?

The mount/volume issues are probably caused by SELinux / other security decisions

I recently took the plunge in a couple of my CI pipelines and had little trouble. The only thing that actually broke was that the `--cache-from` option doesn't accept a tag, where it does for docker. Everything else pretty much Just Worked.

That said, my pipelines are fairly simple, as are my Dockerfiles, so...

It shaved significant time off my pipelines, though, not having to wait for a docker-in-docker service to spin up (I use Gitlab with Kubernetes runners).

I've had a good experience with Podman as a Docker alternative. The Podman CLI is designed to be API-compatible with Docker's CLI, so most things should just work. The only thing I've had to change with my Dockerfiles (or Containerfiles) is the `SHELL` instruction, since Podman by default builds OCI images, and that instruction is not supported by the OCI standard. But you can still specify the shell individually for the `RUN` instructions, or even just use `--format=docker` if you don't want OCI images.

Honestly when I dug into the podman/buildah code it was just differently cursed. It was reallly disappointing.

> Also, buildx won't commit the intermediate layers during the build. So if something fails, you can't just grab the previous intermediate layer and do `docker run --entrypoint /bin/bash` on that layer to poke around.

`buildg debug` (Dockerfile debugger based on BuildKit) to rescue: https://github.com/ktock/buildg