Binary libraries don't help so much -- many libraries use generic types and functions (those are instantiated and compiled for each specific instance) which can only be delivered as library "metadata", not codegen'ed, and the latter is the very slow part.
Wrong since C++11 introduced external templates for common instatiation types.
Additionally, it is not like every C++ library is full of meta-programming templates.
Finally, there are other languages with library "metadata" and generic data types, which can easily beat Rust compile times like Delphi, Eiffel and Ada.
Delphi and Eiffel compilers don't do anywhere near the same level of optimization as LLVM does.
And are you sure GCC's Ada has better compile times? My sense is that so few people use Ada at scale that we don't really know.
It still doesn't do incremental linking, and cargo still builds everything from scratch without any support for binary libraries.