Re:Making AMD GPUs competitive for LLM inference
by FloatArtifact on Tuesday December 24, 2024 @03:25AM
(#42500507)
What kind of model learn and what's its token output on intel gpu's?
Re:Making AMD GPUs competitive for LLM inference
by jeroenhd on Tuesday December 24, 2024 @05:14PM
(#42505309)
If CUDA counts as “just works”, I dread to see the dark, unholy rituals you need to invoke to get ROCm to work. I have spent too many hours browsing the Nvidia forums for obscure error codes and driver messages to ever consider updating my CUDA install and every time I reboot my desktop for an update I dread having to do it all over again.
1. It just works. When i tried to build things on Intel Arcs, i spent way more hours bikeshedding ipex and driver issues than developing
2. LLMs seem to have more cuda code in their training data. I can leverage claude and 4o to help me build things with cuda, but trying to get them to help me do the same things on ipex just doesn't work.
I'd very much love a translation layer for Cuda, like a dxvk or wine equivalent.
Would save a lot of money since Arc gpus are in the bargain bin and nvidia cloud servers are double the price of AMD.
As it stands now, my dual Intel Arc rig is now just a llama.cpp inference server for the family to use.