The problem starts when Rust throws a ton of IR at LLVM. The current focus is on adding optimizations in Rust at higher level (MIR) so that it would cause less work for LLVM.
There is another major problem, which is that rustc/cargo do not do a good job of compiling crates in parallel when one depends on the other.
There may well be other problems too.
On the other side, imagine you can even reduce the "ton of IR" tenfold but if the compilation is sequential, the tenfold speedup is fixed even if you have 32 or 320 cores.
In general, allowing for parallel compilation is much more important topic, and it's worth even redesigning language to make that more achievable.
Which also doesn't mean that producing "ton of IR" is a good thing. But working on allowing maximum parallelism is more fundamental. Also, discovering algorithms to allow all compiler passes as effective as possible is important for something to reach wider usability. But Rust must also get to achieve its promises of being "safer" in the scenarios where C is otherwise used, and I time and again see a lot of "unsafe" keywords in the sources wherever I see some implementation of anything fundamental.
On the other hand, reductions in the amount of computation needed mean you get a faster build for the same resources, or a similar speed build for cheaper.
Depending on your circumstances, one of these things (speed vs. cost) might be much more important than the other. We cannot forget about either.