Micron Kicks Off Production of HBM3E Memory

Re:Micron Kicks Off Production of HBM3E Memory – AnandTech by omneity on Wednesday February 28, 2024 @06:47AM (#39536675)

That checks out in principle, but given that P40 doesn't support NVLink, I wouldn't count too much on using six of them together in a performant manner.

But yeah the best option remains an MI300 if you can afford that.

[ Parent | on HN ]

Re:Micron Kicks Off Production of HBM3E Memory – AnandTech by a_wild_dandan on Wednesday February 28, 2024 @10:23AM (#39539108)

Yeah, my M2 MacBook has 96GB @400GB/s. For $4k or so, it feels like cheating. Does it beat 4x24GB NVIDIA cards? Absolutely not! It's slower and occasionally runs into CUDA-moat software issues. But the capability to daily drive Mixtral 8x7 locally, with great token speeds, is phenomenal.

[ Parent | on HN ]

Re:Micron Kicks Off Production of HBM3E Memory – AnandTech by AnthonyMouse on Wednesday February 28, 2024 @02:49PM (#39542653)

NVLink is most needed for training. For inference a lot of the popular models can usefully be run on multiple GPUs without it:

https://www.reddit.com/r/LocalLLaMA/comments/142rm0m/llamacp...

[ Parent | on HN ]