Regarding prefix sum, the specific question I'm interested in is that SPIRV OpControlBarrier with device scope gets translated into threadgroup_barrier(mem_device) [2]. That's insufficient to make decoupled look-back work. Conversely, if you're not using decoupled look-back, you're not getting the full throughput on GPUs that do support that barrier. I'm wondering how your infrastructure deals with that.
[1]: https://developer.nvidia.com/blog/programming-tensor-cores-c...
[2]: https://github.com/linebender/piet-gpu/blob/d81e5cb4ee145abd...
I'll leave the prefix sum for other devs who know more :D
https://github.com/NVIDIA/cutlass/blob/master/examples/27_am...
//part of nod.ai/shark team
Reading between the lines a little, it sounds like your infrastructure is potentially able to exploit a good deal of the available throughput for FP32 workloads. That's great, and I'm happy to see it! However, for workloads that don't need that much precision, the tradeoff might be a lot less advantageous to M1. That may change again if and when Apple opens up lower-level APIs to their hardware, or reverse engineering delivers usable results.
These are the matmul sizes for the BERT training workload https://github.com/mmperf/mmperf/blob/main/benchmark_sizes/b...
Yes we use the latest MoltenVK (1.3.204.0) installed in the system.
I will let @noxa and other IREE devs chime in on the SPIR-V path but we do support prefix sums etc in the GPU path.
//part of nod.ai team.