Making AMD GPUs competitive for LLM inference

Re:Making AMD GPUs competitive for LLM inference by claytonjy on Monday December 23, 2024 @11:54PM (#42499820)

The cuda situation is definitely better. The nvidia struggles are now with the higher-level software they’re pushing (triton, tensor-llm, riva, etc), tools that are the most performant option when they work, but a garbage developer experience when you step outside the golden path

[ Parent | on HN ]

Re:Making AMD GPUs competitive for LLM inference by cameron_b on Tuesday December 24, 2024 @12:07PM (#42503178)

I want to double-down on this statement, and call attention to the competitive nature of it. Specifically, I have recently tried to set up Triton on arm hardware. One might presume Nvidia would give attention to an architecture they develop, but the way forward is not easy. For some version of Ubuntu, you might have the correct version of python ( usually older than packaged ) but current LTS is out of luck for guidance or packages.

https://github.com/triton-lang/triton/issues/4978

[ Parent | on HN ]

Re:Making AMD GPUs competitive for LLM inference by claytonjy on Tuesday December 24, 2024 @07:16PM (#42505945)

I think you’ve mixed up your Triton’s; I’m talking about Triton Inference Server from NVIDIA while you’re talking about Triton the CUDA replacement from OpenAI

[ Parent | on HN ]