Re:Llama.cpp: Full CUDA GPU Acceleration
by jlokier on Wednesday June 14, 2023 @09:43AM
(#36325816)
The Apple M2 Ultra memory bandwidth is 800GB/s so it's not a long way off 1TB/s.
A100 goes up to 2TB/s in the largest configuration, and H100 claims 3TB/s per GPU node. (These figures keep changing with new variants.) But you can buy several Mac Studios for the price :-)
The real use of H100 is for training, as it can pool many GPUs together with a dedicated high bandwidth network. You can't do that with Mac Studios.
A100 goes up to 2TB/s in the largest configuration, and H100 claims 3TB/s per GPU node. (These figures keep changing with new variants.) But you can buy several Mac Studios for the price :-)
The real use of H100 is for training, as it can pool many GPUs together with a dedicated high bandwidth network. You can't do that with Mac Studios.