GDDR memory buses are so fast that the RAM chips have to be packed tightly around the GPU core to maintain signal integrity, so the limit is more or less how many chips they can physically fit multiplied by the biggest chip capacity their suppliers can provide.

But theoretically RAM chips do not need to be synchronous with each other. Even more, the data lanes on the chip do not need to be synchronous - you can treat each lane as an independent serial channel. And GDDR latency is high enough that longer lanes won't change anything.

You can't quite treat each lane as an independent serial channel; DRAM chips are at least 8 bits wide, and this GPU needs to either connect 16 bits to each die or connect groups of 32 bits to two dies, and the DRAM die does want to get the whole word on the same clock cycle. There's no SERDES at either end like you get with PCIe or Ethernet. Just 512 PHYs doing 32+Gbps PAM3. If you want those to be long-reach PHYs, you're not going to have much die space or power budget left for compute.

Does that mean it's perfectly feasible to have more if one accepts a higher latency? Seems like there could be plenty of use cases where that's preferable.

GDDR chips already have very high latency.

Not exactly. The name of the game with GDDR memory is "speed on the cheap." To do this, it uses a parallel bus with data rates pushed to the max. Not much headroom for things that could compromise signal integrity like socketed parts, or even board traces longer than they absolutely need to be. That's why the DRAM modules are close to the GPU and they're always soldered down.

Also, the latency with GDDR7 is pretty terrible. It uses PAM3 signaling with a cursed packet encoding scheme. At least they were nice enough to add in a static data scrambler this time around! The lack of RLL was kind of a pain in GDDR6.

Like the gtx 970 with 3,5 + 0,4 memory.

Also limited by heat dissipation.

Apple sells a Mac Studio with the M3 Ultra chip and 512GB VRAM (unified memory between CPU and GPU). It costs $9,500.

Their secret is that the memory is manufactured within the chip package.

Why NVIDIA cannot manufacture 512 Gb chips and put 16 of them on the board?

Apples architecture comes with its own trade-offs, it gives them huge capacity and pretty good bandwidth, but not nearly as much as Nvidia's architectures have. The M3 Ultra is 800GB/sec, the RTX 5090 is 1.8TB/sec, and the H200 is 4.8TB/s(!). Huge capacity with middling bandwidth is in vogue because it's a good fit for AI inference, but AI training and most other applications of GPUs need as much bandwidth as they can get.

Well, if you have 16 M3-equivalent chips you can multiply the bandwidth by 16, right? Also, as I understand, ML is basically matrix multiplication and it has O(N³) operations on O(N²) numbers, so bandwidth might be not as important as number of ALUs.

It's actually not within the chip's package. It's soldered to the board. It's just regular, fairly high spec LPDDR5X IIRC, there are just a TON of memory channels.

It's not in the package? TIL... My misunderstanding seems to be common across the interwebs.

I remember Apple used to show slides depicting the M1 SoC as one unit containing a CPU, GPU, Neural Engine, cache, and DRAM all together. But slides shown at an Apple event definitely qualify for artistic license.

The memory is soldered on top of the package, here's an actual real life photo of the M1 package with two LPDDR packages sitting on it:

https://en.wikipedia.org/wiki/Apple_M1#/media/File:Mac_Mini_...

You could call that on-package, but it's not on-package in the same way that GPU HBM is, with that the main die and memory dies are packaged together on the same substrate. That's a much more difficult and expensive process, apparently packaging is the main bottleneck for H100/H200 production.

https://i.imgur.com/hFTPyjk.jpeg

For that H100 image, how do they get the top surface of all the chips on the same plane? Do they solder them "upside down", so the chips are sitting on a reference plane, and the PCB is allowed to "float"? Or is the soldering controlled enough that they can just heat and be done?

Nor is it a chiplet, in which case it would qualify as "in-SoC".

Wait till you learn that unified memory was in common use by pc's long before Apple "invented" it.

The statement is correct; it's not on the substrate similar to AMD's 3D cache or it doesn't use interposer like HBM.

You can consider it like a small PCB that has the CPU die and the memory soldered very closely nearby (and like mentioned +memory channels)

LPDDR5X ~550GB/s vs GDDR7 which is ~1.8 TB/s

This doesn’t really answer the parent’s question.

That the memory is on a PCB close to the CPU/GPU certainly helps with signal integrity, but it is not by any means relevant here. The Apple platform has high memory bandwidth compared to x86 PCs because the CPU has a wide memory bus. You can get similar memory bandwidth out of high-end Epyc and Xeon CPUs which use standard DIMMs but with many more memory channels than a regular desktop computer.

Bandwidth/GPU real estate in terms of area. Biggest GDDR7 chips are 3GB 32-bit, it has 512bit wide bus. And even this is going to moonlight as a space heater

Why Apple packs something like 64 Gb on the CPU chip and NVIDIA cannot?

Apple uses LPDDR4 which comes in densities of up to 16Gb AFAICT. So they can have ~5X as much memory using the same bus width.

Apple uses LPDDR5X, not LPDDR4, iirc

Of course it's ddr5 not 4.

From PCB perspective, LPDDR5(X) interface is quite different from regular DDR5. Same with DDR4 and LPDDR4. Source: have designed few boards with different memory interfaces.

LPDDR dies can stack but GDDR cannot?

What's the usecase of 512 GB memory that you cannot achieve through multi GPUs? Maybe you can make it bit cheaper as you don't require multiple chip, but I would say it is just a maybe because the chip is not the costliest part for Nvidia to manufacture, it is the memory[1].

[1]: https://www.nextplatform.com/2024/02/27/he-who-can-pay-top-d...

One card with twice the memory would let you run the model in half as many cards, at half the speed.

While there are definitely physical limits, the core limitation here is greed. They would sell less cards. Same reason why their consumer cards are limited to ridiculously low amounts of VRAM (16GB on an RTX5080, only 8 on the RTX4060, etc) so if you want to do any serious AI you have to buy their overpriced enterprise cards.

Not just less cards; by making their cards have less ram, they are helping prop up cloud-based inference, which in turn generates revenue for their most expensive line. It is the reason you used to be able to get 12GB ram on a 3060, and that now you have to move up 3 increasingly expensive models to get the same, restricted only by drivers and not capability. They made it clear that this was all intentional because they didn’t want consumer hardware in data centers as it costs profits.

Planned obsolescence, got to sell 7xxx cards somehow.

Wiring also, chips for ddr7 look like they have a lot of pads. And you need pads on the graphics chip for them too.