News for hackers. Stuff that matters.

A comment on Llama.cpp: Full CUDA GPU Acceleration. Parent comment

Re:Llama.cpp: Full CUDA GPU Acceleration by smoldesu on Monday June 12, 2023 @11:30PM (#36304821)
Can hardware vendors even put their differences aside to build such a thing? We can't even build a unified open raster graphics API, and now you're asking for machine learning acceleration in that vein?
[ Parent | on HN ]
Re:Llama.cpp: Full CUDA GPU Acceleration by OkayPhysicist on Monday June 12, 2023 @11:34PM (#36304858)
Machine learning would probably be the simpler API. If you can speak Linear Algebra, you're most of the way there.
[ Parent | on HN ]
Re:Llama.cpp: Full CUDA GPU Acceleration by smoldesu on Monday June 12, 2023 @11:41PM (#36304912)
You're right, and it's why projects like the ONNX runtime exist to unify vendor-specific AI accelerators. Covering the basics isn't too hard.

What GP seems to be asking for is an open CUDA replacement, which is kinda like asking someone to fund a Free and Open Source cruise ship to compete with Carnival for you. You'll get somewhere with some effort, luck and good old human intuition, but Nvidia can outspend you 10:1 unless you have funding leverage from FAANG.

[ Parent | on HN ]