Dawn and WebIDL is also an easy way to add GPU support to any application (that can link C code (or use via a lib)). And Google maintains the compiler layer for the GPU frameworks (Metal, DX, Vulkan ...). This is going to be a great leap forward for GPGPU for many apps.
But yeah, AMD/Nvidia are never going offer huge memory pools affordably on dGPUs.
I think some repos have tried splitting things up between the NPU and GPU as well, but they didn't get good performance out of that combination? Not sure why, as the NPU is very low power.
I have been wanting to get a beefier Mac Studio/mini m2 the more
I’m seeing Apple Silicon specific tweaked packages.
But 64G of VRAM is not the same as GPU mem, apples and oranges
AMDs IGP are way less attractive because they use rather slow DDR4/5 memory while the M2 has blazing fast memory integrated in the package.
We're talking about 50 GB/s vs 400 GB/s. Nvidia's A100 has 1000 GB/s.
Memory bandwidth is usually the bottleneck in GPU performance as many kernels are memory-bound (look up the roofline performance model).
The Pro/Max have double/quadruple that bus width. But they are much bigger/more expensive chips.
I can find very little about it (or other 7000 series integrated gpus). Is this usable at all for running LLaMa in some way?
...for $3.5K minimum, according to the Apple website :/
Is there any chance WebGPU could utilize the matrix instructions shipping on newer/future IGPs? I think MLIR can do this through Vulkan, which is how SHARK is so fast in Stable Diffusion on the AMD 7900 series, but I know nothing about webgpu's restrictions or Apache TVM.