This is more of a step-by-step deep-dive into the ways to profile, diagnose and work through the timing and performance problems. Anyone who doesn't know about many of these tools will probably find them useful (or at least useful to know that they exist) in the future.

Besides, for iteration people are probably in debug, not release mode, which the article mentions is initially 19s vs 2m+. As far as I can see the main conclusion seems to be that there was a compiler regression causing the issues, which is fixed in nightly.

I don't think many users would need to go through these steps to diagnose the problem beyond "wait for a newer compiler release".

In general I don't think being slightly slower in a systems language in cases where you've explicitly asked for full optimisations is that much of a problem when there are simple ways to mitigate the feedback loop (e.g. debug compilations). Would I like these things to be faster? Sure. But with the size of the rust projects I use, it's not noticeable slower than the large established C++ projects I have to deal with (that aren't tuned for compilation speed).

Rust debug builds are also unbearably slow. I wouldn't complain that much if it was just a problems of having full optimizations on

For perf sensitive projects, I set opt-level=1 in debug for at least the dependencies.

Edit: I realized you might have meant that debug compiles still take a long time. In that case, obviously this won't help.

> This is more of a step-by-step deep-dive into the ways to profile, diagnose and work through the timing and performance problems.

I'd argue that when it comes to doing work, I only want to deep-dive into my code. Not third party libraries that "vow" to have been tested + performant, let alone the language/build tools themselves.

I'd be interested to hear what system/field/language you use with compilers that come with guarantees that they will never, ever introduce performance regressions for compile times under any circumstances (presumably with a hefty SLA to match), and where the addition of e.g. an extra innocuous header by some other part of your team will never never ever introduce unexpected nonlinear compile times that require investigations of what the compiler is spending times on.

I suspect this only applies to an extremely small portion of real world code and projects (direct hand optimised asm, perhaps something like forth?)

As far as I am aware the only "guarantees" (interpreting "vow" to mean this) with rust are that edition-bound code will continue to compile correctly, modulo bugs which may be introduced/need fixing.

Of course, it feels like rust could do better, the reputation it has hasn't come from nowhere, but, personally, it's a far cry from e.g. the projects I work on, where we are mandated (by inter-company politics) to use a custom SCons atrocity on C++ that takes two minutes to run a no-op build.

Naturally having scons written in Python has nothing to do with it.

I thought this level of entitlement was restricted to open source users, not devs. AFAIK Rust never promised anything re: the compile-time performance of its standard library, and I know of no 3rd party library that does. You link in a 3rd party library, then that's your code now for better or for worse.

I agree, and even as someone who holds Rust in high regard, I’m becoming increasingly frustrated by this. There doesn’t seem to be one clear root of this problem.

Some of it is the language not being designed with ease of compilation in mind, unlike let’s say, to pick an extreme example, Go. Some of it is the tooling and default settings, especially around incremental compilation, linkers etc. People are working on this, but it takes time.

But unfortunately, a lot of the problem is with the ecosystem, as hinted at in the article. There seems to be no limit to the amount of code bloat and compile time complexity that people are willing to accept to win some microbenchmarks. This includes some very popular crates with lots of dependents, like Tokio.

In the C++ world, the popular “boost” library has a similar problem, and I know shops that avoid it, or large parts of it, simply because the compile times would be unbearable. I really hope that parts of the Rust community start to make similar decisions.

Tokio maintainer here. I've actually been spending a bunch of time recently looking in to how we can reduce Tokio's compile-times. I haven't really been able to find any big wins yet, but one thing has been pretty clear from the benchmarks: The number of dependencies of Tokio is not the problem. They all compile pretty fast and can all compile in parallel. Tokio itself takes a lot longer than the dependencies.

(Except for tokio-macros which takes a long time because it depends on syn and quote. Consider disabling it if you don't need it.)

So I've read "syn" and "slow rust compiles" in pretty much every discussion of "slow rust compiles" and finally googled it and... syn is a Rust parser? For use in macros? Because procedural macros operate on tokens and not an AST? Hm. I'm sure there are reasons for how it ended up like this, but it does smell kinda funny.

If I recall the reason is "rustc's representation of the AST is an implementation detail and unstable, thus macro code may not rely on it (and it will not be exposed)". Hence "syn" came along and provided a stable API and format. "quote" is the inverse function, turning an AST back into tokens.

It allows for things like https://docs.rs/typed-html/latest/typed_html/ to exist.

This is quite interesting. Is `syn` just an interface to the Rust compiler's Rust parser or is it a completely separate implementation of the parser that works at compile-time?

Hopefully it's not the latter but then I wonder where the slowness comes from.

it's a completely separate implementation of the parser

I wish that the proc_macro API from the compiler would do that instead.

syn also let's you define your own parsers for non-Rust syntax, way beyond what rustc could provide.

Macros.

This is a major pain. I wish it get more attention (and hopefully: How about a SINGLE way to do macros? Like zig?).

Serde is another killer on compiling speed.

Well, one of the advantages of having two ways to do macros is that macro_rules! macros are much much faster than procedural macros.

Yeah, but then why not make it the only one? One reason is that proc-macros can "see" the syntax outside the point of declaration(or enclosed). This means you must do macro_rules! {struct.... } instead of declare the struct first then forward it.

First of all, Tokio is my favorite async framework, so thank you and the Tokio team! And second, I appreciate that cargo/rustc compiler errors are clear, helpful, and guide me toward cleaner and thread-safe coding where most other languages happily let me create unsafe hidden race bugs all day long with no warning … oh but they compile so much faster! (I don’t care) The slower compile time really doesn’t bother me when I’m getting a more stable end result… but I’m glad the maintainers are working on improving compile time anyway.

To me the extra compile time makes sense when rust is showing me many unsafe coding issues that languages seem to ignore even with their strict warnings enabled.

… and I don’t get all of the complaining about rust. I’ve written code in a lot of languages and rust has its quirks too, as they all do, but I really like it!

> But unfortunately, a lot of the problem is with the ecosystem, as hinted at in the article. There seems to be no limit to the amount of code bloat and compile time complexity that people are willing to accept to win some microbenchmarks. This includes some very popular crates with lots of dependents, like Tokio.

But isn't this the right trade off for something like Tokio which is at the base of applications which expect to be fast? Or else those applications wouldn't bother with async runtimes and what-have-you. There's also the fact that during the dev cycle, you'll compile tokio & friends once and be done with it. And for CI pipelines, there's caching.

I'm however not a professional developer, so I'm curious if I'm missing something.

Yes, a slow compiletime of a dependency is less painful since you don’t do full recompiles that often, but things can spiral out of control if you have say 50 dependencies and many of them are real slow and you are on your old dual core laptop and find it takes hours, and so on.

Hold up, this is some severe goalpost moving. I realize these comments are coming from different people, but this thread started at:

“tribal knowledge required to bypass it, so early in the the life of a programming language, is a really bad sign.”

and within a few replies we ended at (paraphrasing):

“Granted compilation time is incremental, libraries are only compiled once, debug mode is much faster, but it’s really going to be an issue with a huge project on absurdly obsolete hardware.”

As the author of a project that relies on 50+ projects with some massive dependencies (wgpu, Tokio, rayon), yes it can take a good long while to compile from scratch, but it’s not “hours”, and after first compile that’s all paid for going forward.

Honestly this all seems like grasping at one of Rust’s perceived weak points, but really there are so many better criticisms, I don’t know why compile time gets so much attention.

Mainly because not everyone is willing to buy a last generation threadripper to have the same workflow as apt/yum/rpm/NuGet install libXXX-dev + make.

Including me! There’s a lot of daylight between “an old dual core laptop” and a last gen thread ripper. For example, I’m running on a quad core i7 from 2015.

My old dual core laptop from 2009 works just fine with Visual Studio 2022, even though I wasn't the OP.

And long compile times kill the battery on the go, and cargo check hardly helps when writing GUI code.

I don’t think I follow the point you’re trying make. Are you trying to say Rust compile times are a problem for you on your 13 year old laptop in the context of writing GUI code on battery? If so I think that’s such a specific scenario that it’s hard to draw any generally applicable conclusions.

I also own devices like laptop graphics workstations.

The point I am making is Rust only for those with enough income to buy hardware powerful enough to have a usable workflow or for anyone regardless of what hardware they can afford?

As for coding on the go, classical desktops are a dying breed.

Your laptop from 2009 predates even Rust's earliest stable version by 6 years. Including computers that are older than the language itself is completely unreasonable.

That is because the languages are different and choose different tradeoffs. The tradeoffs might make a language inapplicable in your domain.

Rust has always been marketed as a "systems programming language". C++ is Rust's closes competitor in this domain space and it suffers from terribly slow compiling as well. I would say that due to headers, massive portability baggage and a lack of a standard build system, compiling C++ is even worse. And that is a language that has been around for 30+ years.

I agree that build times are important (i.e. PR build takes us 30 minutes which is insane), but sometimes you don't really have a choice. When you do have a choice, then by all means reap the benefits of fast compilation that languages such as Go or C can offer.

I work on a large rust project professionally, and we have tried to make build times a priority for CI. I’m just noting a couple of things that helped us a lot, in case they’re useful.

- sccache is pretty much essential

- beefiness: GitLab’s default runners for example are way too underpowered and made our builds take like 30 minutes (as opposed to 5 on a beefy machine in CircleCI). We’re working on switching to self-hosted runners for GL

- replacing diesel with sqlx saved us a fair bit of time

- caching the cargo home can help. It wasn’t a massive gain relative to sccache, though. Similar story with caching the target directory

- lld linking was another relatively small win, but a win nonetheless

- splitting things up into smaller libraries and binaries (all in one workspace) helped us a fair bit, since building/testing some targets can now totally skip some libraries, but I imagine YMMV

> caching the cargo home can help. It wasn’t a massive gain relative to sccache, though. Similar story with caching the target directory

Can you please clarify this point? Caching how?

Persisting it across jobs, using whatever your CI framework lets you do. We cache this stuff using the workspace cargo.lock checksum as the primary cache key

C++ projects tend to support massively parallel builds, despite what people tend to complain about with header files. The issue here in this article was partly that Rust only compiles crates in parallel, but while people tend to have lots and lots of C++ translation units, they don't have lots and lots of Rust crates. Languages seem to be moving in the wrong direction here :/.

That's a bit misleading though. The problem with header files is that they tend to get (textually) included in lots of compilation units, and hence rebuilt a lot of times. I did an experiment on a codebase at work where turning on unity builds -- so effectively everything is a single c++ file -- resulted in a 2x speedup in real time, and a 12x reduction in 'user' time.

Of course, such approaches lead to slower hot rebuilds. There's trade-offs worthy of a book. But the point is that C++ is only massively parallel by adding a lot of extra work.

This problem could be overcome by reducing the complexity of header files, but that's generally difficult if template-types are used. In general, the solution is to write code that's closer to C than C++ in the header files. In particular using opaque pointers to hide data.

We're actually seeing a very similiar problem in the linked article. The chief problem is monomorphization (~templates) and the solution is to use boxing (~data hiding via opaque pointers).

That matches my experience as well. It takes a nontrivial amount of discipline, but it’s possible to structure a C++ project for fast builds. In theory, having a module system that isn’t based on text substitution should give Rust an advantage, but on the minus side, the crate structure gives you fewer knobs to optimize.

Care to elaborate further on how to best structure a C++ project for fast builds?

Avoid over-templating, don't inline expensive functions, put expensive #includes inside .cpp files instead of headers, use internal linkage whenever possible, etc. Doable in theory, painful to get there in practice if you've already dug yourself into a hole, and you can't get out of said hole without convincing the rest of your team/organization to change their coding practices accordingly.

Here's my edit-compile-run cycle in dev configuration: https://streamable.com/az397y

I use Qt, boost, lots of templates.

- clang as a compiler

- mold as a linker

- PCH (easy with cmake)

- plugin architecture for the software ; plugins are dynamic libraries when developing (everything is linked statically for releases which take of course much longer to build with lto, etc.)

- building on Linux (it's seriously slower on Windows, when using the exact same compiler on the exact same SSD)

Disabling Windows Defender in your build directories can be a serious boost. NTFS is still slow, though.

You can also disable Windows Defender for the build process, for example disabling it for devenv.exe makes Visual Studio builds waaaaaay faster!

Which editor is this?

Qt Creator I believe? Not 100% sure.

It’s a large topic, but the most important goal is to minimize the amount of code that’s transitively included in headers.

That can mean avoiding templates, or (if you’re willing to do the work) using explicit instantiations in one source file only.

Use the PImpl pattern so that users of a class don’t need to know the declarations of its private members.

And whenever reasonable, design your method signatures so that objects are passed by pointer/reference — this means you can work with forward declarations instead of always having to include the full class declaration.

Do all of that only where reasonable. Small headers usually don’t matter much, and avoiding dependencies on headers that users are likely to include anyway (like <string>) doesn’t win anything either.

I rarely touch C(++) but every time I have to compile something a bit more complicated than the simplest of projects it is noticeably slow. Other languages I use, scala and go and rust, have higher initial cost but seem to suffer less when the sources grow.

A recent example, linkerd proxy compiled in a few minutes, while it took me hours to compile envoy proxy.

Then there are the stories in how ridiculously long it takes to build Chrome or the V8 engine. I’m really not sold that real world c(++) projects are fast to build.

I have a vague memory that at a previous job I was building envoy proxy in a build automation pipeline and it regularly OOMed. While troubleshooting I noticed that most of the time and memory was spent compiling one or two generated C++ files; they contained many many many type parameters. When you have to generate templates something is wrong.

To be more precise, rust is a competitor to C++ and not the other way around.

Also, compiling C++ is _not_ worse than compiling rust projects but again the other way around.

Although compilation times are a hog in both of these languages, it is well known that this is actually one of the biggest rust pain points.

In general, long compilation times are mostly attributed by the complexity of things (algorithms) that compilers are trying to do for you in order to produce machine code and not because there's a lack of standardized build system (a bad idea) nor because of existence of headers.

In particular, biggest "offender" in C++ compilation model is almost always due to the usage of template metaprogramming whereas in rust I'd envision it's the compile time guarantees that must be offered through mechanisms such as borrow checker and alike.

I disagree that it's a pain point in C++ or in Rust. You're generally just doing a lot more stuff and shifting the pain to incremental debug builds (so that the slow path doesn't need recompilation) is a perfectly viable solution. Template metaprogramming has never been the pain point in C++ for me because nobody wanted to use templates (after a certain points), huge deps are.

Anecdotally, the slow builds I've experienced in Rust were case sof overzealous Serialize implementation (aka, people adding Serialize to everything, just in case) or deps on large external projects (often in C++).

Coming from C++, where you either have someone who knows how to debug build times, I think the comparison with rust is unfair. cargo-timings is pretty nice and gives you a hint on what to look for.

This article actually shows debugging build times is not that hard. I did that for a personal project (which compiled part of OpenCV) and I'm certainly no ninja. I remember attempting similar tasks in C++ and getting completely lost.

I live and breathe C++ full time for more than a decade now. Compiling C++ _has_ always been pain point and big efficiency killer. Especially if you come from other languages (I don't but I enjoy learning other stuff too). It doesn't matter if you're directly writing your own template code or not because you're always gonna pull it in indirectly, even in most simplistic and inrealistic case when your sole dependency is C++ runtime library.

OTOH debugging C++ build times is no more than recompiling your code with -ftime-trace and inspecting it with ninjatracing, which btw is a great tool.

However, getting the data as you see is not the real challenge here but in how to use this data to optimize your build pipeline while keeping the functionality and not introducing more technical burden. That is very difficult and for many projects questionable if the benefits will outweigh the total cost invested in it.

  In general, long compilation times are mostly attributed by 
  the complexity of things (algorithms) that compilers are 
  trying to do for you in order to produce machine code and 
  not because there's a lack of standardized build system (a 
  bad idea) nor because of existence of headers.

that is not true - most of the project i worked for suffered from header parsing a lot - sometimes more then 50%, in 100k-1MioK lines of code projects many C++ developer tend to think that the code generation is the time consuming part but that is less true - you will see if you start benchmarking the compilation with todays tools available in Clang and VS2019+

I tend to disagree with this. I've optimized build times on number of occasions on different projects, each being in the venue of multi-million LoC, and none of them benefited greatly from employing precompiled headers, which essentially should help alleviate exact this gap.

Running the build with -ftime-trace almost always has shown in my experiments that the bottlenecks are either in exploding number of template instatiations or more often than not in linking time.

Compiler authors can often make compilation faster, but they have to work at it and think carefully through the problem. It has to be a focus not an afterthought.

Years ago the GNAT Ada compiler authors intentionally avoided implementing precompiled headers. They instead focused on very fast lexical analysis (using a small handwritten lexer optimized for lower case letters because that was the usual case). By careful optimization of a small part of the compiler they made the recompilation of headers faster than loading a precompiled header!! More info: https://dwheeler.com/essays/make-it-simple-dewar.html

Maybe the Rust team needs to add timers to their test suite. This kind of big regression in compilation time should have been caught by the test suite.

We track performance: https://perf.rust-lang.org/

I assure you it's not an afterthought. We track performance changes every week, going as far as reverting highly desired features to avoid regressions. The only reason we accept (small) regressions is for correctness fixes. This regression was only partly exercised by our perf test suite. That's how we grow it: see bad behaviour in the wild, add to the suite.

The source of rust compilation slowness usually boil down to excercising a part of the language that has O(n²) behaviour, like match arm evaluation, trait bound evaluation, or huge expansion due to monomorphization or macro expansion, and the resulting huge linking times from that, or the bottleneck that proc-macros introduce (they get compiled before they can get executed, only relevant on fresh compiles).

The regression was (partly) fixed by adding caching: https://github.com/rust-lang/rust/pull/90423/files and if I recall correctly was introduced by a feature change that fixed a bunch of incorrectly rejected deeply nested trait bounds.

Excellent to hear! Don't get me wrong, I like Rust. I'm hoping we can get compilation speeds up over time, though, that is a common complaint.

Thanks for reposting this. I recalled reading here a while ago some posts on why and how this was put into place, but I couldn't find the link.

In general, Ada has a pretty good compilation model. It has a formalized version of C++'s physical design with spec/body separation. It's also defined grammatically to required the context clause with dependencies first before everything else in a compilation unit (subunits or library units). `separate` also allows you to separately compile those things which might change a lot.

The rust compiler doesn’t spend a lot of time doing things like type checking and borrow checking, except in pathological edge cases (like in the article), and these edge cases are quickly fixed (eg this one was a regression that’s already fixed).

Most time tends to be spent in LLVM. That will also go down over time as the quality of emitted LLVM IR improves due to optimizations in rustc itself.

"compiling C++ is even worse"

In my experience, this has been the other way around. C++ gives you many tools to optimise compilation and when one one takes advantage of them (precompiled headers/shared libs/no boost, etc), the edit-compile-test cycle is fairly fast.

Rust is consistently slow to compile and not much idea of how to go about optimising compile speed.

Both languages are snails compared to Go, of-course.

On most corporations C++ third party libraries are consumed as binary dependencies, hence the large compile times aren't as serious as in Rust.

Rust 1.58 has a bug that introduced a compile time regression, and the knowledge required to bypass it is to just switch to the preview of the next release, which is trivial to set up.

So I really don’t know what you are talking about in your comment. What’s hard about this?

> Rust 1.58

Do you mean 1.57?

As soon as I saw the final binary taking an ungodly amount of time I knew what it was going to be, because I hit it myself, in 1.57, using warp.

Though there are also a few separate older regressions (at least one in 1.56, though I gather somewhat less critical / troublesome).

> the knowledge required to bypass it is to just switch to the preview of the next release

Switching to the previous tends to be more reliable and a better idea, really.

Often when building compilers you trade the speed of compilation against the speed of execution[1], and Rust has (traditionally) chosen the speed of executiong over making the compiler fast.

This won't necessarily always be so, there's e.g. a new compiler backend called cranelift in the works, that makes compilations 30%ish faster, and in return the code isn't as optimized - which is good for a faster inner cycle.

https://pingcap.com/blog/rust-compilation-model-calamity#rus...

I'm not disagreeing, but I could cold compile a million lines of Delphi code in a few minutes on a crappy laptop more than a decade ago. Delphi managed to be both fast to compile and a fast language; of course, it's not as sophisticated or complex a language as C++ or Rust. But still...

The third axis in compiler development is safety. The compiler writer’s dilemma can be framed as:

fast execution, safe execution, fast compilation — choose 2.

ADA.

This is a false(ish) dilema.

The problem of Rust is that is made with a C++ mindset. Is VERY hard to make compiler fast if it follow what C/C++ do (Go is the only evidence against!).

Is a death by thousands cut. The syntax is the FIRST and big one. How you chose it will impact all the pipeline. Then the rest...

Syntax parsing is nowhere close to the critical path for Rust, and I suspect there are no languages where it is today.

Ada isn't entirely safe. It does avoid C-string issues using length-based strings, bounds-checking array access and other features like access types instead of pointers, but it is still possible to hold onto an access to dynamically allocated memory after deletion (though you can somewhat mitigate this with smart pointers).

Sure, neither Rust.

But the things that make Rust safe are not that much slower. Macros, Generics, all that syntax, all that lang complexity (ie: Not just the semantics but the way is surfaced) have a bigger impact.

I choose 3, with OCaml :-)

The biggest issue cited in this article is a regression in the type checker, which is an open bug that apparently is just lower priority than whatever else the Rust team is working on instead of ensuring compilation time is fast.

Only once that was worked around did they article really start having to pay attention to the code generation and linking stages, and the issues there seem to mostly be about lack of parallelism caused by the "crates" model combined with link time optimizations and incremental compiles being off by default. You don't need to make a fundamentally faster code generator that makes worse output to deal with this: the performance losses you get from parallel, incremental, and separate compilation techniques are so minimal that only the most serious C++ projects go out of their way to combine translation units and activate LTO.

Contrary to what you say, rust compilation speed is a top priority for rustc developers. Rust compiler times have gone down YoY every year for the past few years. See, eg, https://nnethercote.github.io/2021/11/12/the-rust-compiler-h...

Then this means that the developer ecosystem continuously finds ways to swallow the improvements in speed. :(

> It means the complexity of the problem space is not solved by the language, and the responsibility of solving it is being passed on to the users.

What languages do it best?

JavaScript is a dystopia of stacked bundlers. Python dependencies and deployment is a toxic superfund wasteland.

You’re not wrong that it’s a problem. But as far as I can tell build+deployment is an unsolved problem for non-hobby projects in most, if not all, languages.

Go is known for having really fast compilations.

Almost Pascal levels of fast.

Go is also known for treating developers like cattle, so, I'll take Rust's trade-off, thanks.

OCaml is a powerful, expressive language with generics from day one, and Go-like build speeds.

All that remains is for it to gain a build tool like mix or cargo. Futzing around with esy works but takes more time than I'm willing to regularly invest in it.

Super excited about 5.00 and multicore!

I really go and forth on this. By separating the package manager and build tool, they leave the door open for either to be swapped out (which is what esy does, incidentally). The flexibility comes at a price of complexity, but it usually works well after climbing the learning curve.

I don't disagree and you're correct, the problem is that I'm usually investing 1-2h segments in playing with OCaml once every few weeks. Having 80% of that time swallowed by a build tool leaves a sour taste. :|

I get what you're saying but it applies to people who already work full time with OCaml. For guys like myself who want to periodically dip their toes and make a slow and steady progress... it makes us feel we're not welcome. (Obviously we're not owed a made bed but I'm hopeful that you get my point and you wouldn't interpret it as entitlement.)

A potential improvement: a very opinionated guide to esy, perhaps? I'd love to author such a guide one day (and would be delighted if somebody beats me to it).

I don't take it as entitlement but more as a case of expectations not matching with reality. With 1 to 2 hrs every few weeks, it's very difficult to ramp in a meaningful way with most languages. Unless they're toy languages or unless people have spent large amounts of time/money for those use cases.

Perhaps you would be best off finding a fully set-up Docker image with a working installation of OCaml/opam, and connecting VSCode to that to act as your dev environment. That's how some educators are teaching it nowadays.

Yep, agreed again. I'll invest the proper amount of hours to not only nail an automated project bootstrapping with esy which I'll then proceed to script with bash/zsh. I'll also exercise a few paths like adding an external dependency, splitting the project to a library and binary, being able to use a REPL (`utop` I think?) in the project, etc.

After that's done I'll also document the entire thing. At that point that might make a good candidate for a blog post to enrich the OCaml ecosystem, too.

Only after that point can I play casually with it, that's true. There's always an initial ramp-up price that has to be paid and in my case that price was higher than I wanted. I'm still willing to go through it all because I absolutely love what OCaml is in every way that I've been exposed to so far (with the exception of build tooling as already said).

While I am going through this I can indeed make use of Docker images and see what they do that I failed to do before. Could be very illuminating.

Build times matter a lot. I've spent a lot of my career working to reduce build and test cycles to improve productivity. Rust is the one environment where no matter how much time we threw at the problem it never got better.

The two biggest time savings where accomplished by using a version without the regression bug, and using incremental builds in debug mode for local dev. Bugs can be introduced in any language, so the only gotcha to be fixed is that maybe incremental compile should be the default.

Note that incremental builds are the default for debug builds.

For reasons that are not entirely elaborated in the article, the author is using release builds for local development.

In some verticals the only way to properly exercise and test the resulting binary is in release mode. Think things like DBs, game engines, etc. It's why good debug information in release is needed.

I'm very happy with my overall experience using Rust, but I would never say "slow build times are not a big deal". I can work around them just fine by myself because it's the kind of thing I'm used to doing, but as more of my team (especially more junior members) start using it for projects, the slowness becomes a big deal. And I don't see that as a problem with my team at all — it's very much a problem with Rust. Especially if you're doing the kind of work where you're switching fairly quickly between different projects (bug blitz, cross-cutting platform work, etc.), you simply can't afford to be wasting time trying to work around bad-by-default compiler performance.

That said, I don't think that the level of slowness we're talking about here is actually an intrinsic property of Rust. Specifically, I think there are a handful of strategies that would really improve things:

- For teams, dead-simple out-of-the-box sccache integration, and officially supported Terraform (etc.) projects for setting it up in your corporate cloud environment. Then your "cold" builds for a project you've never touched before could be a lot quicker.

- A set of inbuilt use-case-driven named profiles for Cargo beyond just "debug" and "release", which you could set as the default for your "debug" or "release" builds. I'm thinking things like "prioritise-runtime-performance-and-incremental-build-time" (maybe with a better name) that sets the right kinds of flags to make that work, given that all those things are achievable if you tweak flags manually.

- Using a faster linker by default, e.g. Mold. Although I don't know whether there are portability or other issues here. Maybe it could just be used where possible?

- Binary repositories. I think people are too quick to dismiss these on the grounds that different features enabled on a crate makes it produce a very different output. You could cache the three most common configurations or something.

TL;DR: I think just not a lot of time has gone into tackling this at a high level, because there have been other priorities, and there are whole lot of directions that could be explored that might make a big difference.

> Using a faster linker by default, e.g. Mold. Although I don't know whether there are portability or other issues here.

mold supports precisely one platform: x86_64 Linux. It also doesn't support linker scripts beyond what is necessary to link libc, and given the creator honestly suggested the suckless practice of editing and recompiling the source code as a possible replacement for them I have my doubts that it'll have a satisfactory replacement for, say, the things I'm currently doing. (Admittedly those things are in ARMv4T rather than x86_64, but still.)

mold supports not only x86-64 but also i386 and ARM64. And besides libc, it can link almost all user-land Linux programs already (I tested that by compiling all Gentoo packages with mold).

If you have to use a linker script for kennel development or embedded programming, mold doesn't work for you, though.

Rust seems to have inherited two favorite C++ development process features: long compilation times and incomprehensible error messages that aren't really related to the actual issue (e.g. just how C++ barfed out a bunch of template errors, borrow checker really loves to barf out very obscure error messages when there's an issue with lifetimes).

If you encounter incomprehensible error messages, I would encourage you to file a ticket at http://github.com/rust-lang/rust/issues. We consider them bugs and are diligent about dealing with them.

The worst error message was basically no error message. After I made some big refactoring to my project, cargo would just exit with:

   error: could not compile `the_project`.

And nothing else. No hint of what the problem is.

It then took me a long time to try to comment out code here and there to find out what was causing the error.

When I finally found out the problem, I took the time to fill a bug report. I don't know if anyone looked at it. https://github.com/rust-lang/rust/issues/73026

Thank you for the great report. It indeed seems to have fallen through the cracks, with no one touching it (it's missing the T-compiler tag, which means I didn't triage it) since it was first cathegorized.

I wouldn't call that lack of output representative of the experience of using rustc though, if I'm allowed to let my pride on my work flare up a bit.

Sorry, I did not mean to say that this was representative of the overall experience. On the contrary, I'm usually quite satisfied with the compiler messages. Thanks for your work!

Thank you! The "bad" thing about having high standards is that when it isn't met it is quite jarring, more so than if it were homogeneously bad or meh, and that ends up bring more frustrating.

While compilation time is a well known Rust pain point, Rust error messages are really one of the best part of the language.

Complicated error messages are clearly no a common occurrence unless you rely heavily on meta-programming tricks, and that is not common in Rust, unlike C++.

And the intention is to improve those too. It's hard, and I would prefer crate writers show more restrain than they do, but we have to meet users where they are.

What the hell are you on about?

Saying in it inherited them, just proves you probably didn't use either.

While compilation times are longer than say Go, they definitely never felt too long, compared to say Java.

But your comment around errors takes the cake. Like maybe, if you use some combination of macros expansion and traits it could get confused.

But Rust errors are on par with Elm. They show the line, they show what went wrong and how to fix. This is a far cry from C++, where using even a bit of templates, results in unintelligible mess.

> they definitely never felt too long, compared to say Java.

Woah... now I am the one who have to ask: what the hell are you on about??

I work on Rust and Java projects, if you exclude running tests, Java compiles very very fast compared to pretty much any language... Rust is a lot slower to compile even on the much smaller projects I've worked on... in this post, a very small project (I think it's like 16,000 LoC, it's mentioned in the beginning) was taking over a minute even on hot builds... To get a build to take this long in Java you would need to have several million LoC! Even if you don't use something like Gradle to cache compilation units and just compile everything from scratch with javac.

Seems like you really need to finish reading the article: hot builds take 3s by the time I find the regression that caused pathological compile times.

You mean, by the time you split up the project into a bunch of tiny crates and mention:

> I don't love how the dependency graph looks

After adding this complexity to the project, yes, some of the crates are building in ~3s, if they are the only ones that needed to be rebuilt.

Meanwhile, in a Java project there is none of this artificial crate splitting. Sure, Java projects have other nonsense, but they don't force you to split up projects into tiny pieces to get decent compile times.

It's a tradeoff: if you make the file the language's compilation unit, then you can't have "circular imports" to allow end users divide their code in an ergonomic manner with fewer "arbitrary" restrictions. We can argue all year on whether Rust settled on the right side of that trade-off.

I'm just surprised that no one has (to my knowledge, anyway) tried this even experimentally. Wouldn't the potential build speed improvement be worth at least trying out the one-file-per-crate strategy?

And I've worked on large Java projects.

Things that have like 420+ dependencies (spring boot + some other requirements). I timed the maven clean compile. And it takes around 2min.

Maven is very slow. Gradle and Bazel avoid doing wasteful things like Maven does so generally are a lot faster... if you want to know the speed of compiling Java you should just run `javac`, assuming Maven speed is Java compilation speed is far from correct even if a lot of projects still use Maven to build Java projects.

Use Maven Daemon. (https://github.com/apache/maven-mvnd) . With Maven Daemon, traditional maven builds get a spectacular boost factor that put them ahead of Gradle. (Haven't tried Bazel though)

> Java compiles very very fast compared to pretty much any language

In my experience C# compiles faster, but I still agree with you on all points.

Build times have regressed considerably the past few releases (at least for me). Not that I'm complaining about the tradeoffs, the new features are generally worth it. Whats most annoying is that it's often hard to tell which features are expensive until you've already wasted a bunch of time introducing em.

But I've got to say I'm super jealous of the level of introspection the rust compiler allows here. I'd definitely be willing to change a few C# coding patterns in code bases I maintain to benefit build times, but as is, that's a really painful trial and error process. And sometimes you get lucky finding the root cause, but if you don't... well, it's not a great experience.

Not surprising, considering the chief architect of C# is the guy famous for making Turbo Pascal.

Rust's errors are definitely the best of any language I've used. That's one reason it's nice to write Rust after a day fighting with SQL Server for example which seems to still think in 2021 that "Syntax error" is a reasonable diagnostic message.

As to "How to fix it" however if you have lifetime confusion the error reported by the borrow checker probably isn't going to suggest refactoring your code with a design that makes sense even though that's likely the correct solution. It would be cute if the Rust compiler said "I think you want a Boxed collection of strings here to use the algorithm you're attempting, change how this is represented in all the related structs too, here's how:" but that's unrealistic and, more importantly, no other language is doing a better job here.

While Rust makes the errors themselves look easy, the right suggestion is context sensitive anyway. In my AoC solutions, unwrap() was the correct answer anywhere the problem is "I have an Option or a Result and I wish that I didn't". Because it's AoC if I'm wrong to be so cavalier I will eat a panic immediately, and I don't plan to maintain this stuff. In a real project I probably want to explicitly handle None and Error in some places, and pass on Errors in other places, even when I don't I should write an expect() in case I'm maintaining this years from now.

More likely in lifetime confusion scenarios the equivalent C++ compiles but just has mysterious crashes or bugs and the equivalent Java has runtime Null reference errors, Concurrency errors, a truly enormous memory leak, data races, other mysterious mutation bugs, or all of the above. I'll take a compile time error from the borrow checker thanks.

> It would be cute if the Rust compiler said "I think you want a Boxed collection of strings here to use the algorithm you're attempting, change how this is represented in all the related structs too, here's how:" but that's unrealistic

I think we could provide such messages in some cases. :)

For example, if you have an impl Trait return type and have multiple returned values of different types, we suggest using Box<dyn Trait> when valid. It's just a lot of work to get those in place. You're effectively implementing a language that does those things automatically, plus all the work to print out the actual diagnostic.

I teach C++ and Rust to college students, and their number one praise of Rust over C++ is how much more helpful the error messages are. It’s really night and day, so I can’t even fathom how you would equate the two.

It's easy to fathom once you go beyond the school examples and start working on bigger codebases. The error messages are verbose and simple until you hit cases where borrow checker complains about an error in a completely different part of the codebase than where the actual issue here is. This is where it behaves similar to C++ (and, honestly, some other languages) and there all the verbosity in the world doesn't help.

Have you really used Rust on big codebases? One of the big advantage of Rust is especially that it fails far earlier than C++. Borrow checker error are local error that prevent bad behavior to propagate and cause problem on unexpected part.

Someone has never used Rust

Rust has some of the best compiler error messages out there so not sure what you're talking about there. And C++'s compiler error message issue stopped being an issue since like GCC 4.