Re:AMD MI300X vs. Nvidia H100 LLM Benchmarks
by wkat4242 on Saturday July 20, 2024 @05:57PM
(#41020361)
Yes but as far as i understand it, the interconnect is not really important for model inference. But for model training more so.
[ Parent
| on HN ]
Re:AMD MI300X vs. Nvidia H100 LLM Benchmarks
by spott on Saturday July 20, 2024 @06:13PM
(#41020484)
Depends if you can fit the whole model into vram or not. If you can’t then you need some sort of gpu parallelism, and you need some sort of communication between the different gpus. But maybe that messaging is small enough that it doesn’t majorly slow down inference. I’m not sure.