Making AMD GPUs competitive for LLM inference

Re:Making AMD GPUs competitive for LLM inference by mrcsharp on Tuesday December 24, 2024 @02:53AM (#42500369)

I will only consider AMD GPUs for LLM when I can easily make my AMD GPU available within WSL and Docker on Windows.

For now, it is as if AMD does not exist in this field for me.

[ Parent | on HN ]

Re:Making AMD GPUs competitive for LLM inference by e-max on Tuesday December 24, 2024 @08:53PM (#42506306)

Isn't it already available somehow? I didn't test it seriously, I just needed to quickly run Whisper but

  $ rocminfo  | grep -E "WSL|XTX"
  WSL environment detected.
  Marketing Name:          AMD Radeon RX 7900 XTX

[ Parent | on HN ]

Re:Making AMD GPUs competitive for LLM inference by mrcsharp on Thursday December 26, 2024 @10:08PM (#42519638)

Interesting. Looking here [1] it seems like this is a thing now. Will dive deeper into this later but looks promising.

[ Parent | on HN ]