Llama.cpp: Full CUDA GPU Acceleration

Re:Llama.cpp: Full CUDA GPU Acceleration by wyldfire on Tuesday June 13, 2023 @01:21AM (#36305777)

Well, we could do a much much better job of it but in fact Qualcomm does compete with NVIDIA for use cases like this (inference). Both in mobile devices and the data center.

Disclaimer: I work at Qualcomm.

[ Parent | on HN ]

Re:Llama.cpp: Full CUDA GPU Acceleration by Tostino on Tuesday June 13, 2023 @03:36AM (#36306844)

What is on offer optimized for running these LLMs?

[ Parent | on HN ]

Re:Llama.cpp: Full CUDA GPU Acceleration by wyldfire on Tuesday June 13, 2023 @05:11AM (#36307588)

The hexagon NSP is reasonably well suited for running ML in general. I know it's used for some image/CV use cases and I think it will work well for language models, but maybe suboptimal for the recent large ones.

This processor shows up in both snapdragon SoCs and cloud ai 100.

[ Parent | on HN ]