Safety doesn't cost that much in compilation time. `cargo check` performs full type-checking and borrow checking, and it's reasonably fast already.

Rust's performance depends very much on optimizations, mainly LLVM. It could be faster if you were willing to accept less optimized code, or change coding style to use fewer libraries and fewer abstractions. But if you want the same speed without any compromises, then I don't see any way around that.