We split each service into crates, plus some libraries. There were some native dependencies as well, which could really impact compile times, as well as some codegen for things like protobuf.

From a quick tokei, looks like 49kloc of Rust.

One thing I've observed is that every smallish projects (~10k LoC) can take a while to build if you're working on slow network filesystems.

When using local disks, a 10k LoC project takes ~5 seconds to build

When using network FS, the same project takes ~23 seconds to build

That doesn't seem particularly surprising. Lots of software is slow when your storage is slow.

You're right that some slowdown is expected, but for me personally I hadn't realized how bad this particular FS was, nor had I expected how much it impacted build times

Wouldn't this be true of any compiler? Do other compilers have optimizations that avoid excess disk usage when on a slow file system?

Yes, I think that's right, but I don't have any data to compare rust to other compilers.

I wouldn't be surprised if Rust/Cargo does more disk IO than other build tools, though. Rust does a lot of compile time code gen and caches a lot of stuff on disk.

FWIW we did this to significantly reduce storage usage

    strip = "debuginfo"

This was a ~20x reduction in size, 279M -> 19M. If you have slow storage it could really help performance (don't recall, personally).

Good hint, thank you. I'm trying this out and I do see an improvement in performance. The ~23 second build now takes about ~14 seconds

Hell yeah! I'm so glad I could help, that's a big win.

FWIW it does break debuggers since you won't have the symbols. Just comment it out when you need a build that has all of that info.