I'm not sure what you're implying by this, because the Crystal compiler is doing a lot of work as well. The global type inference is the primary reason for Crystal's compiler speeds, and it's why it's not likely to substantially improve. It's not slow for the sake of being slow.

Java does type inference in a number of places and javac is not slow.

Java doesn't do type inference at the level of Crystal. It's not the presence of type inference, it's the fact that almost everything in a Crystal program can be inferred.

javac produces Java bytecode, not LLVM IR.

javac exists since at least 15 years, Crystal "only" 6 years.

Maybe things will get better with time?

What work would that be?

? The borrow checker which is literally the reason Rust exists.

Borrow checking is almost always very cheap, and lifetime are erased before codegen which is the really slow part.