The way I sometimes think about stuff like this is that it's useful in reverse: you're going to have to write optimised code that doesn't really resemble Java. But if you're in a situation where there is no GPU available, that same code will execute fine on the JVM.

Yeah, that's TornadoVM's big thing. You just write normal code and develop it locally as you would when running on the CPU (most friendly dev environment). And then it just figures out what HW is available and uses it.

This fits really nicely with scaling up on the cloud. If you have a parallel workload that you need to auto-scale very quickly, you could migrate from small CPU to big CPU, to AVX512, to GPUs to FPGAs, all automatically. Maybe it won't produce code that can beat hand-tuned Verilog but it'll do it a lot faster and cheaper.

Re: no objects. That doesn't actually mean no objects. Recall this is based on Graal. Graal is really, really good at scalar replacing objects automatically, i.e. converting allocations into local variables, even when they're being e.g. allocated in a method and then returned. This is how Truffle gets such good performance: it just recursively inlines everything into one giant method and then wires up all the allocation sites to use sites. It turns out you can really use a lot of objects this way, yet at runtime they all disappear.

Given it's based on the same compiler infrastructure I'd expect that to also be true of TornadoVM; perhaps a TornadoVM developer can chime in. Probably you can do method calls and working with wrapper types without hitting the limitation, or at least, it can be added without too much effort (I guess the hard part is flattening the types into arrays as value types, which may require a more sophisticated optimisation pass).

That's correct. TornadoVM specializes the OpenCL code depending on the target device to increase performance. Also, as you pointed out, it supports some objects through the partial escape analysis by applying scalar replacements.

Also, TornadoVM supports method calls and invitations to native code (e.g., Math library). The examples in the presentations are for simplicity but we have some use cases, such as KFusion Kinet with ~7k lines of Java code for computer vision (https://github.com/beehive-lab/kfusion-tornadovm).

> You just write normal code and develop it locally as you would when running on the CPU (most friendly dev environment).

That's OpenCL's big thing. Which TornadoVM is sitting on top of. And OpenCL never really took off, it's in a bit of a weird spot currently being partially deprecated.

It's a fine idea on paper, but there doesn't seem to be much of a market demand for it. Probably because it's not a thing anyone really needs. Step debugging is about the only reason to bother at all, but you really don't need any sort of heterogeneous compute migration system framework thing to pull that off. It's just a local flag or even a re-compile option.

> This fits really nicely with scaling up on the cloud. If you have a parallel workload that you need to auto-scale very quickly, you could migrate from small CPU to big CPU, to AVX512, to GPUs to FPGAs, all automatically.

Scaling up in the cloud is a perfect example of why TornadoVM itself isn't that interesting. Just like you can't write a normal multithreaded single process application and have it magically scale out over a network, nor can you here write a normal CPU process and have it automatically migrate between CPU & GPU. You have to design your program from the ground up around concepts like SIMT, memory coalescing, DMA bandwidth/latency, etc... These aren't just "here's some tricks to go _even faster_" things, these are "you must do this or don't even bother using the GPU at all" things.

To make this work you don't need to port a different language or VM to the platform, you need the compute equivalent of Kubernetes or Hadoop or whatever other cloud scaling infrastructure system you want. Otherwise this largely boils down to a Java to OpenCL transpiler. Cool, but it's not advancing the state of art here, either.

We are integrating TornadoVM with Apache Flink. So the idea is to execute unmodified map/reduce Flink applications automatically on heterogeneous hardware by using TornadoVM in the task-managers. This is part of a current European project (https://e2data.eu/).

Correct, since we employ Graal for parts of the compilation, we also enjoy the benefits of inlining and escape analysis. If an object is replaced by scalars then we can use it on a GPU. Otherwise (if a real object allocation is required) then the compiler will bail out and execution will continue normally on the CPU (with compiled code from Graal or C2).

> when running on the CPU (most friendly dev environment)

It's pretty hard to buy a computer these days that doesn't have some sort of GPU. Either it has on board graphics or it has a board connecting it to a screen.

Back in 2010 I worked on a user-library and preprocessor that did the same thing [1]. Had exactly the same restrictions. Classes are supported if they act as dumb structs.

Recursion is now supported in CUDA via dynamic parallelism (or faking the stack) but it is not performant at all.

The ideal language for GPU programming is closer to Julia or a MATLAB-like language where arrays and matrices are first-class.

[1] https://github.com/AEminium/AeminiumGPU

> The ideal language ..

That pulled up "Chapel" in my head and a couple of clicks later the net pulled this up:

"PGAS (Partitioned Global Address Space) programming models were originally designed to facilitate productive parallel programming at both the intra-node and inter-node levels in homogeneous parallel machines. However, there is a growing need to support accelerators, especially GPU accelerators, in heterogeneous nodes in a cluster. Among high-level PGAS programming languages, Chapel is well suited for this task due to its use of locales and domains to help abstract away low-level details of data and compute mappings for different compute nodes, as well as for different processing units (CPU vs. GPU) within a node. In this paper, we address some of the key limitations of past approaches on mapping Chapel on to GPUs ..."

https://pldi19.sigplan.org/details/CHIUW-2019-papers/4/GPUIt...

> How can you even say you're running Java if you don't have objects or dynamic allocation?

It's not any different from, say, JavaCard, which doesn't even have java.lang.String or garbage collection.

> And when you think about how a GPU works, it completely makes sense. A high level language for a GPU will not look like Java

It can still have Java syntax, which can also leverage existing IDEs and other libraries (eg. for unit testing).

>And when you think about how a GPU works, it completely makes sense. A high level language for a GPU will not look like Java

Can someone explain this further? I am genuinely interested in learning what makes Java etc not suitable for GPUs.

CPUs are like taxis and GPUs are like buses. You have 64 people sitting in your bus all taking the same path. As soon as two people need to arrive at two different destinations (branching) this whole system breaks down. Eventually the bus must drive to all destinations. Then there is the time wasted stopping to load passengers. To efficiently utilize the bus you must load 64 people at the same. If you were to let people hail the bus like a taxi it would constantly have to stop to just load a single passenger while the other 63 people in the bus have to wait for that single person (random memory access).

A pointer can point to object of any arbitrary sub type in any memory location. Suddenly you have both branching and random memory access. If your program is branching heavily then you may end up with 1 piece of data being computed by a core that can handle 64 pieces of data at once and random memory access kills you because GPUs don't have huge caches like CPUs. The way GPUs deal with high latency is that they simply batch a large amount of work and switch to a different "thread" during memory access. If all your threads are busy loading from memory then you won't see any speedups.

Finally GPUs do not actually have 4096 cores. The RX Vega 64 only has 64 cores and those cores only run at around 1.3GHz. A Ryzen with 8 cores running at 4GHz with out of order execution can trivially outperform a GPU that is only using 1.6% (1/64) of its theoretical performance.

Analogies break down or.. is this actually within the analogy? Some buses, they permit small variances from the set route to deal with specific issues which can be done en-route.

The simplest is "drop me off anywhere on the route, but only pick up on demand at scheduled stops"

The more complex is "wear one diversion every now and then"

Some small buses actually self-optimize to a locally efficient route. Post-Buses in rural locations.

It is surely possible GPU can segment, into small disjoint sets of "like" and so continue to offer parallelism, but at reduced intensity? Or, have point inefficiencies in the computation, toward the overall goal of "all alike"

Yes. GPUs also do not have tires. Who cares?

Other way around - GPUs are not suitable for Java.

Java is a dynamically-allocated, garbage-collected language. So far as I know, GPU code doesn't have the ability to dynamically allocate memory. (I mean, I guess you could, but you'd probably need one memory pool per parallel computational unit. Otherwise, you'd need a global lock on the global memory pool.)

Java is a general programming language. GPUs are not general-purpose processors.

That's off the top of my head. There may be other reasons as well.

CUDA can dynamically allocate, more recente versions even support unified shared memory blocks with the host processor.

It is also possible to do native memory allocation in Java.

Imposing restrictions on a Java subset is hardly any different of imposing restrictions on C and C++, which apparently is ok to do.

The big difference that makes Java cumbersome is that you always have object references (ie pointers), thus an array of objects is really an array of pointers to objects that are in another memory location.

C/C++ structs(classes) on the other hand represents a memory layout so an array of "objects" would be laid out as a single contigious block of memory. (and this works well for a GPU)

There is/was a propsal for Java for value types but it hasn't yet been approved. https://openjdk.java.net/jeps/169

(As a side note, C# classes are used as refernces like in Java but they also have "struct" in C# that behaves like C structs, and this is what the Unity boost compiler leverages and could be used for a GPU variant with less restrictions)

Not necessarily, although the language does not guarantee it, flattening to structs does happen in some JITs.

Grated it isn't perfect, hence the value types proposal.

Here is one of the CUDA commercial offerings for Java and .NET,

https://developer.nvidia.com/altimesh-hybridizer

> Not necessarily, although the language does not guarantee it, flattening to structs does happen in some JITs.

Only if escape analysis says it didn't go anywhere. Which works for iterators but it's not going to work for input & output arrays to this system, especially since it's asynchronous meaning it did escape the stack frame.

I've never understood why modern GPUs don't address the programmability restrictions with a minimal CPU core on die. While one wouldn't choose a GPU for programs best written dynamically or recursively, having to block on the CPU for simple statements seems like an addressable bottleneck.

GPU programs run fast exactly because programs are written with these limitations in mind. GPUs trade stuff like branch prediction for data parallelism. You can't just have both on the same die and achieve the same throughput.

Using dynamic memory liberally, having lots of indirect branches and non-local memory access - that's pretty bad for performance on the CPU as well, even though it is optimized for that. It's just that people don't notice because they're used to programming that way.

> I've never understood why modern GPUs don't address the programmability restrictions with a minimal CPU core on die.

They basically do. There's a minimal "core" that does things like branching and that core manages a bunch of threads that all execute the same instruction in parallel. But there's a lot of those minimal cores still, so transistors spent on it need to be worth it. Spending transistors to make a nicer programming model means fewer transistors spent going faster.

Alternatively the design you're looking for is Intel's Xeon Phi, which died. Intel's Xe looks maybe more traditional GPU in architecture but TBD, it might be less restrictive in this regard. Maybe.

I mean, without all that stuff, is it even still Java? Is it even something more than just a few text-replacements on identically-mapped syntax elements?