I'm not willing to even know about the HW at all, the higher level my code the more opportunities for the JIT to optimize my code.
What I really want is something like Mathematica that can JIT to GPU.
As another commenter mentioned all the API's assume you're a discrete GPU off the end of a slow bus, without shared memory. I would kill for an APU that could freely allocate memory for GPU or CPU and change ownership with the speed of a pagefault or kernel transition.
It genuinely deserves to exist alongside pytorch. It's not just Google's latest framework that you're forced to use to target TPUs.
For "embarrassingly parallel" jobs vector extensions start to eat tiny bits of the GPU pie.
Unfortunately, just slapping thousands of cores works poorly in practice. You quickly get into the synchronization wall caused by unified memory. GPUs cleverly work around this issue by using numerous tricks often hidden behind extremely complex drivers (IIRC CUDA exposes some of this complexity).
The future may be in a more explicit NUMA, i.e. in the "network of cores". Such hardware would expose a lot of cores with their own private memory (explicit caches, if you will) and you would need to explicitly transact with the bigger global memory. But, unfortunately, programming for such hardware would be much harder (especially if code has to be universal enough to target different specs), so I don't have high hopes for such paradigm to become massively popular.
IMHO, the real issue is cache coherence. GPUs are spared from doing a lot of extra work by relaxing coherence guarantees quite a bit.
Regarding the vendor situation - that's basically how most of computing hardware is, save for the PC platform. And this exception is due to Microsoft successfully commoditizing their complements (which caused quite some woe on the software side back then).
I too am very interested in this model. The Linux kernel supports up to 4,096 cores [1] on a single machine. In practice, you can rent a c7a.metal-48xl [2] instance on AWS EC2 with 192 vCPU cores. As for programming models, I personally find the Java Streams API [3] extremely versatile for many programming workloads. It effectively gives a linear speedup on serial streams for free (with some caveats). If you need something more sophisticated, you can look into OpenMP [4], an API for shared-memory parallelization.
I agree it is time for some new ideas in this space.
[1]: https://www.phoronix.com/news/Perf-Support-2048-To-4096-Core...
[2]: https://aws.amazon.com/ec2/instance-types/c7a/
[3]: https://docs.oracle.com/en/java/javase/24/docs/api/java.base...
To me it feels somewhat like programming for the segmented memory model with its near and far pointers, back in the old days. What a nightmare.
(That was idealized - actually there may be ~5 alternative driver APIs even on a single platform each with its own strengths)
I find needing to write seperate code in a different language annoying but the UX of it is very explicit of what is happening in the memory which is very useful. With really high performance compute across multiple cores ensuring you don't get arbitrary cache misses is a pain. If we could address CPUs like we address current GPUs( well you can but it's not generally done) it would make it much much simpler.
Want to alter something in parallel, copy it to memory allocated to a specific core which is guaranteed to only be addressed by that core and the do the operations on it.
To do that currently you need to be pedantic about alignment and manually indicate thread affinity to the scheduler etc. Which ia entirely as annoying as GPU programming.
- You need to compile shader source/bytecode at runtime; you can't just "run" a program.
- On NUMA/discrete, the GPU cannot just manipulate the data structures the CPU already has; gotta copy the whole thing over. And you better design an algorithm that does not require immediate synchronization between the two.
- You need to synchronize data access between CPU-GPU and GPU workloads.
- You need to deal with bad and confusing APIs because there is no standardization of the underlying hardware.
- You need to deal with a combinatorial turd explosion of configurations. HW vendors want to protect their turd, so drivers and specs are behind fairly tight gates. OS vendors also want to protect their turd and refuse even the software API standard altogether. And then the tooling also sucks.
What I would like is a CPU with a highly parallel array of "worker cores" all addressing the same memory and speaking the same goddamn language that the CPU does. But maybe that is an inherently crappy architecture for reasons that are beyond my basic hardware knowledge.