It really depends on what you're actually asking for. Do you want, for example, a "slimmer" Rust compiler that jettisons all the stuff that supports older language editions? Do you want a "simpler" compiler that only uses straightforward algorithms at the expense of compiler speed? Or do you want a faster compiler that does less optimization at the expense of code generation quality? The latter, at least, is something that the Cranelift backend for rustc hopes to achieve.
The borrow checker is technically optional (as proven by mrustc), but if you wanted to implement it, it's no longer a set of simple scope rules, but more like a flavor of Prolog inside the compiler.
These things are awesome for the power and usability of the language, but aren't tiny.
https://github.com/thepowersgang/mrustc
But AIUI that is not aimed at being a Rust compiler you would actually use day-to-day, just at being a way to bootstrap a Rust toolchain without needing a Rust compiler.
[0] https://tiny.cc/