We did our builds in docker, for various reasons. So we relied on the docker buildx cache and some other tricks that I don't recall because I didn't work a ton on the build system.

Oh... Docker-based Rust builds are extremely slow in my experience.

I seem to remember that caching was really, really bad.

Docker isn't the issue. I just did a clean build, 19 minutes 44 seconds (with a ~year old compiler though).

Just out of curiosity, can you check the time for the same project with a more recent toolchain?

    Finished dev [optimized + debuginfo] target(s) in 16m 58s

Neat.