Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel
by GeekyBear on Wednesday October 11, 2023 @06:46PM
(#37851340)
> IDK about ML part, but equivalent performing Ryzen mini pc cost me 3x less over m1 macbook
When running an ML workload, the Nvidea A100 has massive GPU compute resources, and a large amount of GPU local high bandwidth memory, so it's ideal, but is nowhere near low cost.
A consumer Ryzen chip is inexpensive, but lacks in both memory bandwidth and GPU resources.
The M2 Ultra has access to way more RAM than consumer GPUs, many times the memory bandwidth of a Ryzen (800GB/s vs Ryzen 7 1800X at 40 GB/s) with a large amount of local GPU resources.
Even stepping up to a Threadripper Pro would only get you a quarter of the memory bandwidth, and those aren't exactly cheap either.
When running an ML workload, the Nvidea A100 has massive GPU compute resources, and a large amount of GPU local high bandwidth memory, so it's ideal, but is nowhere near low cost.
A consumer Ryzen chip is inexpensive, but lacks in both memory bandwidth and GPU resources.
The M2 Ultra has access to way more RAM than consumer GPUs, many times the memory bandwidth of a Ryzen (800GB/s vs Ryzen 7 1800X at 40 GB/s) with a large amount of local GPU resources.
Even stepping up to a Threadripper Pro would only get you a quarter of the memory bandwidth, and those aren't exactly cheap either.