Re:How to Run Llama 3 405B on Home Devices? Build AI Cluster
by adrian_b on Monday July 29, 2024 @10:58AM
(#41100977)
I have replied to the posting claiming that an Epyc might be better, which it is not.
The bandwidth per core is the same for a 16-core Ryzen and for a 96-core Epyc.
Of course if you compare it with a GPU, the GPU wins in memory bandwidth. However the GPUs are useless for FP64 computations, unless you are a big corporation that can afford the huge prices of the "datacenter" GPUs.
For ML/AI, GPUs are the normal choice, except when you cannot afford those with enough memory, when you may still want to fall back to cheap CPUs, as argued in the parent article.
A $1500 Ryzen 9 9950X system will have a double price in comparison with a mini-PC, but it will be 3 to 4 times faster in any program whose performance is dominated by the speed of the array operations. This speed ratio is true for the fastest CPU that could be put now in a mini-PC, i.e. AMD Strix Point (a HX 370 does 192 FP32 FMA per clock cycle, while a 9950X does 512 FP32 FMA per clock cycle and its clock frequency is much higher; it should be noted that these values are similar to those for integrated GPUs; the best iGPUs have a double number of ALUs, but their clock frequency is much lower, slightly more than half of that of a CPU). In comparison with a low-power Intel CPU or older AMD CPU, the speed advantage of 9950X would be even greater.
The bandwidth per core is the same for a 16-core Ryzen and for a 96-core Epyc.
Of course if you compare it with a GPU, the GPU wins in memory bandwidth. However the GPUs are useless for FP64 computations, unless you are a big corporation that can afford the huge prices of the "datacenter" GPUs.
For ML/AI, GPUs are the normal choice, except when you cannot afford those with enough memory, when you may still want to fall back to cheap CPUs, as argued in the parent article.
A $1500 Ryzen 9 9950X system will have a double price in comparison with a mini-PC, but it will be 3 to 4 times faster in any program whose performance is dominated by the speed of the array operations. This speed ratio is true for the fastest CPU that could be put now in a mini-PC, i.e. AMD Strix Point (a HX 370 does 192 FP32 FMA per clock cycle, while a 9950X does 512 FP32 FMA per clock cycle and its clock frequency is much higher; it should be noted that these values are similar to those for integrated GPUs; the best iGPUs have a double number of ALUs, but their clock frequency is much lower, slightly more than half of that of a CPU). In comparison with a low-power Intel CPU or older AMD CPU, the speed advantage of 9950X would be even greater.