News for hackers. Stuff that matters.

A comment on Llama.cpp: Full CUDA GPU Acceleration. Parent comment

Re:Llama.cpp: Full CUDA GPU Acceleration by Tostino on Tuesday June 13, 2023 @03:36AM (#36306844)
What is on offer optimized for running these LLMs?
[ Parent | on HN ]
Re:Llama.cpp: Full CUDA GPU Acceleration by wyldfire on Tuesday June 13, 2023 @05:11AM (#36307588)
The hexagon NSP is reasonably well suited for running ML in general. I know it's used for some image/CV use cases and I think it will work well for language models, but maybe suboptimal for the recent large ones.

This processor shows up in both snapdragon SoCs and cloud ai 100.

[ Parent | on HN ]