If you examine what takes time in the Rust compiler, the borrow checker is never a significant component.

The largest single bit tends to be codegen, which is why cargo check is so much faster than cargo build.