That is not really true. Yes, it would help first compiles a bit, but only the initial compiles. That's a one-time improvement, that while important, doesn't solve the big problems.
With Rust is almost impossible, just to check a random project without investing the required time to wait for its build to finish.
Also maybe I am special, but it is quite typical to switch project branches quite often, or just get libraries from another project.
All of which I can easily do via binary libraries stagging.
After you first cargo build, you have all of the dependencies pre-compiled. Every single "cargo build" after that's compile time would be completely unaffected by pre-built dependencies. You could try sccache; builds are still slow. Most people care about those secondary builds, not about the initial build, so telling them that this solves their problem is simply not true. It's not downplaying it, as I said, the first builds are also important. Unfortunately, it's just not that simple.
If solving Rust builds were as easy as this, we'd have just done it long ago. We care tremendously about improving compile times. It is a constant request on surveys. We put a lot of engineering work into this.
So while it may seem like bashing, it is more trying to be a positive critic of something that real matters and is a show stopper for some environments.
In any case I look forward to any improvement in build times, and an eventual story regarding binary libraries.
A debug backend or interpreted based one, e.g. ocaml, GHC hugi, could help quite substantially.