Only if they were specifically written for GPUs. GPUs are incredibly limited. Expecting to run arbitrary code on them is only going to disappoint you. If you could just run regular code on GPUs then everyone would have done so. Programming language support is not a barrier. The GPU itself is. In fact, you should consider yourself lucky if you have problems that don't need GPUs.
Sure, no arbitrary loops though,..but a constrained subset will work (see kernel abstractions.jl)
IMO this is worth very little; as long as there is a heavily optimized version written in C/assembly (ie MKL), the fact that Julia might be faster than standard-but-slower versions like OpenBLAS doesn't actually gain me anything as a user.
Mostly the bits about how you have to workaround the Julia GC.
So it doesn't seem like a great option for libraries that might have lots of different bindings, where C/C++/Fortran (and maybe Rust?) seem to be the only reasonable choices.
Julia is now experimenting with pure-Julia code that is faster than many LAPACK/BLAS implementations. But I actually care little about the fact that they are a bit faster. Same with the new ODE package in Julia, which is unsurpassed in terms of functionality. The actual reason these pure-Julia packages excite me is that they are generic enough to run relatively easily on different hardware (GPUs for instance), they are on top of modern compilers (LLVM), and most importantly, they are written in a language that permits much higher level of interoperability.
To rephrase it much more simply: The fact that you can not use scipy special functions efficiently inside of any of the autodiff frameworks is exemplary of how incredibly limiting python is, when it comes to developing new numerical techniques. Julia does not have that problem as far as we know.