I work on a large rust project professionally, and we have tried to make build times a priority for CI. I’m just noting a couple of things that helped us a lot, in case they’re useful.

- sccache is pretty much essential

- beefiness: GitLab’s default runners for example are way too underpowered and made our builds take like 30 minutes (as opposed to 5 on a beefy machine in CircleCI). We’re working on switching to self-hosted runners for GL

- replacing diesel with sqlx saved us a fair bit of time

- caching the cargo home can help. It wasn’t a massive gain relative to sccache, though. Similar story with caching the target directory

- lld linking was another relatively small win, but a win nonetheless

- splitting things up into smaller libraries and binaries (all in one workspace) helped us a fair bit, since building/testing some targets can now totally skip some libraries, but I imagine YMMV

> caching the cargo home can help. It wasn’t a massive gain relative to sccache, though. Similar story with caching the target directory

Can you please clarify this point? Caching how?

Persisting it across jobs, using whatever your CI framework lets you do. We cache this stuff using the workspace cargo.lock checksum as the primary cache key

C++ projects tend to support massively parallel builds, despite what people tend to complain about with header files. The issue here in this article was partly that Rust only compiles crates in parallel, but while people tend to have lots and lots of C++ translation units, they don't have lots and lots of Rust crates. Languages seem to be moving in the wrong direction here :/.

That's a bit misleading though. The problem with header files is that they tend to get (textually) included in lots of compilation units, and hence rebuilt a lot of times. I did an experiment on a codebase at work where turning on unity builds -- so effectively everything is a single c++ file -- resulted in a 2x speedup in real time, and a 12x reduction in 'user' time.

Of course, such approaches lead to slower hot rebuilds. There's trade-offs worthy of a book. But the point is that C++ is only massively parallel by adding a lot of extra work.

This problem could be overcome by reducing the complexity of header files, but that's generally difficult if template-types are used. In general, the solution is to write code that's closer to C than C++ in the header files. In particular using opaque pointers to hide data.

We're actually seeing a very similiar problem in the linked article. The chief problem is monomorphization (~templates) and the solution is to use boxing (~data hiding via opaque pointers).

That matches my experience as well. It takes a nontrivial amount of discipline, but it’s possible to structure a C++ project for fast builds. In theory, having a module system that isn’t based on text substitution should give Rust an advantage, but on the minus side, the crate structure gives you fewer knobs to optimize.

Care to elaborate further on how to best structure a C++ project for fast builds?

Avoid over-templating, don't inline expensive functions, put expensive #includes inside .cpp files instead of headers, use internal linkage whenever possible, etc. Doable in theory, painful to get there in practice if you've already dug yourself into a hole, and you can't get out of said hole without convincing the rest of your team/organization to change their coding practices accordingly.

Here's my edit-compile-run cycle in dev configuration: https://streamable.com/az397y

I use Qt, boost, lots of templates.

- clang as a compiler

- mold as a linker

- PCH (easy with cmake)

- plugin architecture for the software ; plugins are dynamic libraries when developing (everything is linked statically for releases which take of course much longer to build with lto, etc.)

- building on Linux (it's seriously slower on Windows, when using the exact same compiler on the exact same SSD)

Disabling Windows Defender in your build directories can be a serious boost. NTFS is still slow, though.

You can also disable Windows Defender for the build process, for example disabling it for devenv.exe makes Visual Studio builds waaaaaay faster!

Which editor is this?

Qt Creator I believe? Not 100% sure.

It’s a large topic, but the most important goal is to minimize the amount of code that’s transitively included in headers.

That can mean avoiding templates, or (if you’re willing to do the work) using explicit instantiations in one source file only.

Use the PImpl pattern so that users of a class don’t need to know the declarations of its private members.

And whenever reasonable, design your method signatures so that objects are passed by pointer/reference — this means you can work with forward declarations instead of always having to include the full class declaration.

Do all of that only where reasonable. Small headers usually don’t matter much, and avoiding dependencies on headers that users are likely to include anyway (like <string>) doesn’t win anything either.

I rarely touch C(++) but every time I have to compile something a bit more complicated than the simplest of projects it is noticeably slow. Other languages I use, scala and go and rust, have higher initial cost but seem to suffer less when the sources grow.

A recent example, linkerd proxy compiled in a few minutes, while it took me hours to compile envoy proxy.

Then there are the stories in how ridiculously long it takes to build Chrome or the V8 engine. I’m really not sold that real world c(++) projects are fast to build.

I have a vague memory that at a previous job I was building envoy proxy in a build automation pipeline and it regularly OOMed. While troubleshooting I noticed that most of the time and memory was spent compiling one or two generated C++ files; they contained many many many type parameters. When you have to generate templates something is wrong.

To be more precise, rust is a competitor to C++ and not the other way around.

Also, compiling C++ is _not_ worse than compiling rust projects but again the other way around.

Although compilation times are a hog in both of these languages, it is well known that this is actually one of the biggest rust pain points.

In general, long compilation times are mostly attributed by the complexity of things (algorithms) that compilers are trying to do for you in order to produce machine code and not because there's a lack of standardized build system (a bad idea) nor because of existence of headers.

In particular, biggest "offender" in C++ compilation model is almost always due to the usage of template metaprogramming whereas in rust I'd envision it's the compile time guarantees that must be offered through mechanisms such as borrow checker and alike.

I disagree that it's a pain point in C++ or in Rust. You're generally just doing a lot more stuff and shifting the pain to incremental debug builds (so that the slow path doesn't need recompilation) is a perfectly viable solution. Template metaprogramming has never been the pain point in C++ for me because nobody wanted to use templates (after a certain points), huge deps are.

Anecdotally, the slow builds I've experienced in Rust were case sof overzealous Serialize implementation (aka, people adding Serialize to everything, just in case) or deps on large external projects (often in C++).

Coming from C++, where you either have someone who knows how to debug build times, I think the comparison with rust is unfair. cargo-timings is pretty nice and gives you a hint on what to look for.

This article actually shows debugging build times is not that hard. I did that for a personal project (which compiled part of OpenCV) and I'm certainly no ninja. I remember attempting similar tasks in C++ and getting completely lost.

I live and breathe C++ full time for more than a decade now. Compiling C++ _has_ always been pain point and big efficiency killer. Especially if you come from other languages (I don't but I enjoy learning other stuff too). It doesn't matter if you're directly writing your own template code or not because you're always gonna pull it in indirectly, even in most simplistic and inrealistic case when your sole dependency is C++ runtime library.

OTOH debugging C++ build times is no more than recompiling your code with -ftime-trace and inspecting it with ninjatracing, which btw is a great tool.

However, getting the data as you see is not the real challenge here but in how to use this data to optimize your build pipeline while keeping the functionality and not introducing more technical burden. That is very difficult and for many projects questionable if the benefits will outweigh the total cost invested in it.

  In general, long compilation times are mostly attributed by 
  the complexity of things (algorithms) that compilers are 
  trying to do for you in order to produce machine code and 
  not because there's a lack of standardized build system (a 
  bad idea) nor because of existence of headers.

that is not true - most of the project i worked for suffered from header parsing a lot - sometimes more then 50%, in 100k-1MioK lines of code projects many C++ developer tend to think that the code generation is the time consuming part but that is less true - you will see if you start benchmarking the compilation with todays tools available in Clang and VS2019+

I tend to disagree with this. I've optimized build times on number of occasions on different projects, each being in the venue of multi-million LoC, and none of them benefited greatly from employing precompiled headers, which essentially should help alleviate exact this gap.

Running the build with -ftime-trace almost always has shown in my experiments that the bottlenecks are either in exploding number of template instatiations or more often than not in linking time.

Compiler authors can often make compilation faster, but they have to work at it and think carefully through the problem. It has to be a focus not an afterthought.

Years ago the GNAT Ada compiler authors intentionally avoided implementing precompiled headers. They instead focused on very fast lexical analysis (using a small handwritten lexer optimized for lower case letters because that was the usual case). By careful optimization of a small part of the compiler they made the recompilation of headers faster than loading a precompiled header!! More info: https://dwheeler.com/essays/make-it-simple-dewar.html

Maybe the Rust team needs to add timers to their test suite. This kind of big regression in compilation time should have been caught by the test suite.

We track performance: https://perf.rust-lang.org/

I assure you it's not an afterthought. We track performance changes every week, going as far as reverting highly desired features to avoid regressions. The only reason we accept (small) regressions is for correctness fixes. This regression was only partly exercised by our perf test suite. That's how we grow it: see bad behaviour in the wild, add to the suite.

The source of rust compilation slowness usually boil down to excercising a part of the language that has O(n²) behaviour, like match arm evaluation, trait bound evaluation, or huge expansion due to monomorphization or macro expansion, and the resulting huge linking times from that, or the bottleneck that proc-macros introduce (they get compiled before they can get executed, only relevant on fresh compiles).

The regression was (partly) fixed by adding caching: https://github.com/rust-lang/rust/pull/90423/files and if I recall correctly was introduced by a feature change that fixed a bunch of incorrectly rejected deeply nested trait bounds.

Excellent to hear! Don't get me wrong, I like Rust. I'm hoping we can get compilation speeds up over time, though, that is a common complaint.

Thanks for reposting this. I recalled reading here a while ago some posts on why and how this was put into place, but I couldn't find the link.

In general, Ada has a pretty good compilation model. It has a formalized version of C++'s physical design with spec/body separation. It's also defined grammatically to required the context clause with dependencies first before everything else in a compilation unit (subunits or library units). `separate` also allows you to separately compile those things which might change a lot.

The rust compiler doesn’t spend a lot of time doing things like type checking and borrow checking, except in pathological edge cases (like in the article), and these edge cases are quickly fixed (eg this one was a regression that’s already fixed).

Most time tends to be spent in LLVM. That will also go down over time as the quality of emitted LLVM IR improves due to optimizations in rustc itself.

"compiling C++ is even worse"

In my experience, this has been the other way around. C++ gives you many tools to optimise compilation and when one one takes advantage of them (precompiled headers/shared libs/no boost, etc), the edit-compile-test cycle is fairly fast.

Rust is consistently slow to compile and not much idea of how to go about optimising compile speed.

Both languages are snails compared to Go, of-course.

On most corporations C++ third party libraries are consumed as binary dependencies, hence the large compile times aren't as serious as in Rust.