Intel announces Arc B-series "Battlemage" discrete graphics with Linux support

Re:Intel announces Arc B-series "Battlemage" discrete graphi... by numeri on Tuesday December 03, 2024 @04:39PM (#42311831)

GPU inference is always a balancing act, trying to avoid bottlenecks on memory bandwidth (loading data from the GPU's global memory/VRAM to the much smaller internal shared memory, where it can be used for calculations) and compute (once the values are loaded).

Splitting the model up between several GPUs would add a third much worse bottleneck – memory bandwidth between the GPUs. No matter how well you connect them, it'll be slower than transfer within a single GPU.

Still, the fact that you can fit an 8× larger GPU might be worth it to you. It's a trade-off that's almost universally made while training LLMs (sometimes even with the model split down both its width and length), but is much less attractive for inference.

[ Parent | on HN ]

Re:Intel announces Arc B-series "Battlemage" discrete graphi... by amelius on Tuesday December 03, 2024 @05:29PM (#42312303)

> Splitting the model up between several GPUs would add a third much worse bottleneck – memory bandwidth between the GPUs.

What if you allowed the system to only have a shared memory between every neighboring pair of GPUs?

Would that make sense for an LLM?

[ Parent | on HN ]

Re:Intel announces Arc B-series "Battlemage" discrete graphi... by imtringued on Wednesday December 04, 2024 @06:03AM (#42316359)

At least for LLMs and transformers this isn't relevant. Having 8x the chips and 8x the memory bandwidth is always better. Interchip communication for matrix multiplication against a constant left matrix with a tiny right matrix isn't bandwidth bound, only latency bound.

[ Parent | on HN ]