Instead of waiting for the future maybe you could look to the past? That's how graphics cards used to work back a couple decades ago.

AMD Radeon Pro SSG had 4 nvme slots on the card itself but that was 2017 but with direct storage API that might be able to have some gains for large models.

I could never get a solid answer wether that was presented as memory to the GPU or just as a PCIE switch with NVME drives hanging off one side and the GPU on another.

As far as I remember it was presented as a drive and was good for sequential read but you had to use AMD's API to get the full benefit

PCIe switch with NVMe drives and the GPU behind it, they actually just are presented straight to windows even.

https://www.youtube.com/watch?v=-fEjoJO4lEM

In principle they could be used with an API like DirectStorage RDMA or CUDA GPUDirect RDMA (which dates back to Kepler) and in this case they would never need to talk to the CPU, given appropriate software support. But it's not going to be presented as GPU memory ever, it's going to work like a block storage device you can do RDMA requests against, most likely.

https://docs.nvidia.com/cuda/gpudirect-rdma/

https://developer.download.nvidia.com/video/gputechconf/gtc/...

Now technically - it all depends on what you mean by "as GPU memory" because PCIe is all RDMA anyway, even CPU-to-GPU is a RDMA operation. That's why there's the whole thing about "resizable BAR" etc - that's the aperture window in CPU memory that gets mapped in from the GPU memory.

So technically yes you can map those SSDs in "as GPU memory" via GPUDirect RDMA block storage (or DirectStorage), but you can do that with a regular NVMe SSD in an adapter card too. The SSG is just a "combo GPU+SSD card" in the same way QNAP makes those "combo network+SSD cards", but with a lot of fanfare/marketing around it.

To be absolutely fair, Fiji/Vega is a good design for that since it doesn't have a bunch of memory packages around it. But it wasn't what AMD trumpeted it as, as the LTT video describes, it was a very specific reaction to the question of 'workstation GPUs are using a lot of memory, HBM2 can't be scaled like that, how do we put more memory on a Fiji/Vega GPU for workstation users". And AMD's claims that HBM meant you could just swap everything around and not have to worry about framebuffer size were never true, the PCIe bus itself is not fast enough for that.

https://www.youtube.com/watch?v=NQ8YlXh-jbI&t=280s

Honestly nobody can really give a straight answer to why that isn't the case with HBCC on Vega either... swapping has always been a thing, why is HBCC different from swapping in general?

The bandwidth and latency is as much a bottleneck as the capacity is. That's why recent ML and GPU chips have moved to on-package "high-bandwidth memory." Even if you were to add more high-bandwidth memory off-package (via lots of parallel DIMMs, say), it's difficult to actually manage this memory well, because of a host of competing factors, including a) on-chip resources (controllers etc) that must consume area and power to manage the new memory, b) increased design cost for validating handling the extra (crappier than HBM) memory, c) software not knowing how to hide the (much higher) latency of this memory - deep learning isn't as good at memory hierarchies as CPUs are, d) a chip that has spare on-chip bus bandwidth for this kind of thing has wasted bus bandwidth. Et cetera.

So if you are building GPUs or AI accelerators, you tend to just go ahead and build this in.

It might not be as fast but still allow you to train a large model in a reasonable amount of time, I suppose.

Not going to happen:

https://www.servethehome.com/wp-content/uploads/2022/08/NVID...

That unless an utterly revolutionary new interconnect technology comes...

LPDDR is getting adopted more and more too, with CPUs losing memory expansion capabilities in exchange of huge power savings.

I bet a lot of people would be happy if their GPU could train large models, even if it took 5x as long.

That's getting a multi-level DRAM config, see Grace Superchip for an example.

Or just a plain slower GPU: Apple's GPU series, but that's at very much higher price tags than desktop GPUs at a given perf level.