If the spec requires that the compiler look for a certain error and produce a diagnostic, then the compiler has to spend time doing that. The more complicated the spec is, the more work the compiler has to do to find the errors.

For example, Rust is frequently said to have long compile times. There are a number of interesting reasons why that is often true, but if we ignore the pathological cases then what we find is that borrow checking takes a significant fraction of that compile time. This is a big trade–off between features and complexity that the Rust language made very deliberately: the advantages of the borrowing rules are what makes Rust such a great language. The cpu time spent checking that those rules have been followed are a small price to pay, but not a negligible one.

Is there a single such which you cannot disable by compiler flags?

You cannot disable the borrow checker in Rust with a compiler flag. You might as well try to turn off the type checker, or ask it to ignore syntax errors.