AMD's MI300X Outperforms Nvidia's H100 for LLM Inference

Re:AMD's MI300X Outperforms Nvidia's H100 for LLM Inference by fleischhauf on Thursday June 13, 2024 @08:32AM (#40668861)

AMDs deep learning libraries are very bad the last time I checked, nobody uses amd in that space for that reason. Nvidia has a quazi monopoly, that's the main reason for the price difference IMHO.

[ Parent | on HN ]

Re:AMD's MI300X Outperforms Nvidia's H100 for LLM Inference by tracker1 on Thursday June 13, 2024 @10:12AM (#40669908)

I think their best option might be more/better prosumer options at the higher end of consumer pricing. Getting more hobbyists into play just on the value proposition.

[ Parent | on HN ]

Re:AMD's MI300X Outperforms Nvidia's H100 for LLM Inference by More-nitors on Thursday June 13, 2024 @09:37AM (#40669485)

this...

nearly 95% of deeplearning github repos are "tested using cuda gpu - others, not so sure"

the only way to out-run nvidia is to have 3~10x better bang-for-buck.

Or AMD can just provide a "DIY unlimited gpu RAM upgrade" kit -- a lot of people are buying macstudio 128gb ram because of its "bigger ram-for-buck" than nvidia gpus

[ Parent | on HN ]

Re:AMD's MI300X Outperforms Nvidia's H100 for LLM Inference by sroussey on Thursday June 13, 2024 @11:21AM (#40670773)

I heard apple m4 ultra using 256gb HBM for studio and pro, but I don’t buy it. The 256GB maybe. But a HBM memory control that would go unused on laptops doesn’t pass the smell test.

[ Parent | on HN ]

Re:AMD's MI300X Outperforms Nvidia's H100 for LLM Inference by nabla9 on Thursday June 13, 2024 @08:40AM (#40668941)

> that's the main reason for the price difference IMHO.

Explain why the performance difference does not matter?

AMD does only 33% better with a chip that has 2X transistors and 2X memory.

[ Parent | on HN ]