As others have pointed out most of the performance hit is in the optimization of the relatively verbose IR code the Rust compiler sends to LLVM, not the Rust compiler itself.

There's been multiple passes at making the Rust compiler faster. It'll happen. There's a GCC-based Rust implementation in the works too.