The reason is AMD and Nvidia don't is that they don't want to cannibalize their high end AI market. Intel doesn't have a high end AI market to protect.

There are products like this one: https://www.intel.com/content/www/us/en/products/sku/232592/...

As far as I understand it, it gives you 64 GiB of HBM per socket.

that's a CPU. We are talking about GPUs here for highly parallel matrix multiplication problems. 2 different beasts.

This CPU is a bit of a hybrid because it as integrated matrix multiplication accelerators: https://en.wikipedia.org/wiki/Advanced_Matrix_Extensions

NVidia and AMD make $$$ on datacenter GPUs so it makes sense they don't want to discount their own high-end. Intel has nothing there so they can happily go for commodization of AI hardware like what Meta did when releasing LLaMA to the wild.

Is nVidia or AmD offering 128gb cards in any configuration?

They aren't "cards" but MI300x has 192GB and MI325x has 256GB.

You can run an AMD APU with 128GB of shared RAM.

you can do that with nvidia too but it takes you from 6tok/s to 6s/token or worse (not even exaggerating)

It's too slow and not very compatible. Most BIOSes also don't allow sharing that much memory with GPU (max like 16GB).

Isn’t that setting just a historical thing annd ann integrated GPU is able to access any system memory that is mapped by the IOMMU? I assume this is how it works for people using the NVIDIA Jetson AGX Orin 64GB Developer Kit to do inference. I do not know why it would be different for AMD APUs.

I remember somebody complaining about it on reddit, unable to overcome some BIOS limitation on an AMD G processor. Even on M3 Max one had to issue a special command to enable GPU to access more memory.

The command on the M3 Max is a sysctl command to adjust an operating system enforced limit. That is different than the aperture setting in the bios. The limitation on AMD is more interesting and more relevant.