We see folks using our linalg library for this and deep learning: http://nd4j.org We maintain our own c++ and cuda stack underneath this as well. It also allow control of these native components from java. We implement everything from our own garbage collector for cpu and gpu to our own cuda kernels.