As of 2002 (JDK 1.4) Java has excellent integration with native memory (you can freely pass pointers from C/FORTRAN to Java and vice versa[1]).
There are numerous Java math libraries that use BLAS/LAPACK already[2]. In fact, AFAIK, most Java matrix math libraries use FORTRAN code (at least as an option).
[1]: Java side: http://docs.oracle.com/javase/7/docs/api/java/nio/ByteBuffer... C side: http://docs.oracle.com/javase/7/docs/technotes/guides/jni/sp...
[2]: For example, https://github.com/fommil/matrix-toolkits-java, http://mikiobraun.github.io/jblas/
I had used memory mapped buffers (which should be equivalent in performance), and there was no way to make the JIT inline access to these arrays. It was all calls (indirect, not properly branch predicted at that). Equivalent code to C++, running 10 times slower, with no way to speed it up.
(And the reason I was using memory maps, if you insist -is a 2GB read-only dataset used by multiple processes at the same time - I went to C++ eventually, because there was no way to get reasonable performance from Java, either memory use or speed. This is circa 2010)
For example, the Java Chronicle library[1] uses these techniques, as well as memory mapped files, to implement fast persistent message queues.
The sun.misc.Unsafe class is used extensively by JDK classes, and is meant for internal use. It provides intrinsics that are translated to a single machine instruction. Normally, you don't use the class directly. For example, you use, say, AtomicInt for CAS operations (which, internally uses s.m.Unsaafe) or the ByteBuffer class (which internally uses s.m.Unsafe for direct pointer access). The JDK classes add all sorts of protection (like range checks) around s.m.Unsafe, but if you know what you're doing, using s.m.Unsafe directly and eschewing some of those protections (usually adding ones more pertinent to your domain), you get some performance gains which may be significant depending on your use case.
(Personally, when programming Java I find it more convenient to use primitive arrays as opposed to matrix libraries, but that might be dependent on the operations I tend to do: lots of increment/decrements and only occasional linear algebra. I guess this isn't exactly relevant to the R replacement question.)
Direct byte buffers are very common in high-performance Java code. Reading/writing from/to those buffers can be made just as fast as plain Java arrays.
We've been working hard and now systematically to get Renjin (also R on the JVM) to run CRAN packages: http://packages.renjin.org. Renjin also compiles C and Fortran code to JVM bytecode, though there is still some work to do there as well.
Regarding the hand-tuned matrix math libraries, there's nothing to stop you from using them with Renjin - you can drop in MKL or Atlas as desired, or fall back to pure-Java versions in a pinch.
As for Java, Oracle together with AMD, are in the process of making the GPU trasparent to Java developers as part of the Sumatra project.
So this is one are where R could benefit of running on Oracle's JVM. It remains to be seen if other Java vendors would adopt such feature.
There are many other R implementation efforts going on right now -- Radford Neal lists a few (as well as his own) here: http://radfordneal.wordpress.com/2013/07/24/deferred-evaluat...
The presentation focuses on the R programming language, which they nicely show has all sorts of misfeatures that impede rapid execution. If you're going to not try to have compatibility with R and CRAN, you might as well start from scratch with design and performance in mind, as in Julia: http://julialang.org/