Even native compilers like clang will use LLVM, and LLVM IR can look a lot like JVM bytecode.

The VM abstraction is not a barrier to fully optimizing for your architecture, but rather how much time you can spend converting IR/bytecode into assembly, whether you do it at runtime, and whether runtime information lets you optimize even further.