They've got a lot of idiosyncrasies at the moment, some of it working around the fact ADD some/directory/ could never be cached (so my build scripts maintain some.directory.tar.gz and those are ADDed instead), but I really should. I guess I'd put it under my github profile (I'm also csirac2 there).

There's two types of ARM builds, ones which can cross-compile and those that can't. The ones which can't cross-compile are done with qemu-binfmts and we chroot into an ARM filesystem and run the build there.

Perhaps the only useful contribution would be the fact that I persist the ccache up to the docker host with a shared ccache volume, and that helps enormously especially for the qemu-binfmts builds which can be quite slow.

Where does your original (non-qemu) toolchain come from? Is that built inside or outside of a container?

(em)debian provides nearly enough of what I need most of the time. The trigger for going to a ARM chroot is when I can't get build dependencies installed properly on an amd64 host. Either that or the thing I'm compiling just isn't developed to be cross-compiler friendly and it's too much work to hotwire it to be so.

For example, say I need libfoo, I have an amd64 host (being the docker container). Sometimes I just can't get the libfoo:armhf or libfoo-dev:armhf package installed because it would break/conflict with the amd64 host's version of it in some way. xapt often helps but then sometimes screws up by re-packaging something that has an "all" arch (non-arch-specific) to something armhf specific (eg. foo-data). This ultimately either conflicts with the host or fails to be named properly in such a way that it meets the build-deps of the project.

Sometimes I know it would be easier in some cases to avoid the debian packaging ecosystem, but for my workflow and distribution requirements it brings a lot of benefits.

Edit: see here https://wiki.debian.org/EmdebianToolchain