News for hackers. Stuff that matters.

A comment on AMD Inference. Parent comment

Re:AMD Inference by andyferris on Wednesday October 02, 2024 @06:33PM (#41725547)
Wait, looking at that link I don't see how it avoids downloading CUDA or ROCM. Do you use MLIR to compile to GPU without using the vendor provided tooling at all?
[ Parent | on HN ]
Re:AMD Inference by steeve on Thursday October 03, 2024 @02:24PM (#41733399)
We do use ROCm and CUDA. Only we sandbox it with the model and download only the needed parts which are about 1/10th of the size.
[ Parent | on HN ]