AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

Re:AMD's MI300X Outperforms Nvidia's H100 for LLM Inference by yobbo on Thursday June 13, 2024 @07:23AM (#40668243)

The model code is comparatively tiny compared to pytorch or CUDA itself. Translating models from CUDA/C could be laborious but not a barrier.

Making AMD work effortlessly with pytorch et al should make the switch transparent.

[ Parent | on HN ]

Re:AMD's MI300X Outperforms Nvidia's H100 for LLM Inference by mistymountains on Thursday June 13, 2024 @11:25AM (#40670818)

These kinds of comments make me think few people have actually tried. My experience has been 1 work day of getting things set up to work the same as before for training and testing (PyTorch).

[ Parent | on HN ]

Re:AMD's MI300X Outperforms Nvidia's H100 for LLM Inference by imtringued on Thursday June 13, 2024 @01:14PM (#40672163)

You have to consider that the average person who tried to do machine learning on AMD GPUs got burned in the past decade and has no reason to change their opinion. Also in the past it was much harder to get access to cutting edge GPUs from AMD. The fact that AMD drops GPU support for ROCm quickly also earns them scorn. I don't think it is an unfair assessment. They earned their reputation.

[ Parent | on HN ]

Re:AMD's MI300X Outperforms Nvidia's H100 for LLM Inference by muxr on Sunday June 16, 2024 @06:47PM (#40700743)

ROCm has improved a lot. And you can rent mi300x in the cloud now. So if you have a program that runs on Nvidia GPUs, it takes no time to test it on a cloud mi300x. If it works you can use it and save some money in the process.

[ Parent | on HN ]

Re:AMD's MI300X Outperforms Nvidia's H100 for LLM Inference by zombiwoof on Wednesday June 19, 2024 @02:48AM (#40725539)

AMD supports PyTorch out of the box , these comments make me feel no one has tried or even working on this stuff

[ Parent | on HN ]