News for hackers. Stuff that matters.

A comment on Llama.cpp: Full CUDA GPU Acceleration. Parent comment

Re:Llama.cpp: Full CUDA GPU Acceleration by eurekin on Tuesday June 13, 2023 @05:50AM (#36307852)
Is that RAM as fast as A100/H100 VRAM? AFAIR, gpus push ~ 1 TB/s ish
[ Parent | on HN ]
Re:Llama.cpp: Full CUDA GPU Acceleration by jlokier on Wednesday June 14, 2023 @09:43AM (#36325816)
The Apple M2 Ultra memory bandwidth is 800GB/s so it's not a long way off 1TB/s.

A100 goes up to 2TB/s in the largest configuration, and H100 claims 3TB/s per GPU node. (These figures keep changing with new variants.) But you can buy several Mac Studios for the price :-)

The real use of H100 is for training, as it can pool many GPUs together with a dedicated high bandwidth network. You can't do that with Mac Studios.

[ Parent | on HN ]
Re:Llama.cpp: Full CUDA GPU Acceleration by eurekin on Thursday June 15, 2023 @02:22PM (#36344568)
That figures are really impressive. A single node is still useful for finetuning smaller models. Apple could move the needle in this market a bit
[ Parent | on HN ]