Re:Rust – Faster compilation with the parallel front-end in ...
by insanitybit on Friday November 10, 2023 @10:35AM
(#38220138)
It was over a year ago so it's a bit hard to recall... Maybe 20 minutes clean? 1 or 2 minutes for cached. Things have probably improved since then but idk. We did stuff like protobuf, we had a few proc-macros of our own, plenty of serde, and I think ~3 native dependencies (zstd, librdkafka, something else I don't recall). The native dependencies could be brutal, it caused long serial stalls, if I recall correctly. Linking took up a lot of time as well, but I forget why we didn't use mold, there was a reason at the time... but, again, over a year ago.
Here's the code: https://github.com/grapl-security/grapl/tree/main/src/rust
Re:Rust – Faster compilation with the parallel front-end in ...
by Dowwie on Friday November 10, 2023 @12:12PM
(#38221449)
what was the caching strategy?
Re:Rust – Faster compilation with the parallel front-end in ...
by insanitybit on Friday November 10, 2023 @12:28PM
(#38221702)
We did our builds in docker, for various reasons. So we relied on the docker buildx cache and some other tricks that I don't recall because I didn't work a ton on the build system.
Re:Rust – Faster compilation with the parallel front-end in ...
by Yoric on Friday November 10, 2023 @03:45PM
(#38224281)
Oh... Docker-based Rust builds are extremely slow in my experience.
I seem to remember that caching was really, really bad.
Re:Rust – Faster compilation with the parallel front-end in ...
by insanitybit on Friday November 10, 2023 @03:49PM
(#38224339)
Docker isn't the issue. I just did a clean build, 19 minutes 44 seconds (with a ~year old compiler though).
Re:Rust – Faster compilation with the parallel front-end in ...
by estebank on Friday November 10, 2023 @03:58PM
(#38224426)
Just out of curiosity, can you check the time for the same project with a more recent toolchain?
Re:Rust – Faster compilation with the parallel front-end in ...
by insanitybit on Friday November 10, 2023 @09:39PM
(#38227147)
Finished dev [optimized + debuginfo] target(s) in 16m 58s
Neat.
Re:Rust – Faster compilation with the parallel front-end in ...
by thinkharderdev on Friday November 10, 2023 @11:10AM
(#38220580)
Depends on the code and whether you are doing a release or debug build. I work on a very large rust project (~1m LoC) with a lot of dependencies. We've split it into multiple crates and the compile times don't really frustrate my dev workflow (incremental compilation works well and debug builds are pretty fast anyway). But building in our CI pipeline where we do a fully optimized build (single codegen unit, LTO enabled) it takes a while (~30m) which is annoying when you are waiting for a hotfix to be ready. It's also incredibly resource intensive (mainly linking with LTO enabled) so we've been the bane of our platform teams existence since we need something like 50GB of memory in our build container to do a full release build :)
Don't hesitate to be brutally honest — nothing can shock me as my go-to compiler is GHC.