Partly, and the other part is that it degrades allocation performance for the majority of non-problematic programs, which is what most people actually focus on.

But if we're being fair, latency of tracing GC isn't a problem for most programs either. So latency is largely a red herring, except when it's not, and you had better know when it's not, regardless of whether you're using C/C++/Rust or a runtime with tracing GC.