News for hackers. Stuff that matters.
Re:LLaMa running at 5 tokens/second on a Pixel 6 by Havoc on Wednesday March 15, 2023 @01:06PM (#35171371)
Any more details? I'm guessing they're leveraging the NPU in the pixel?
Re:LLaMa running at 5 tokens/second on a Pixel 6 by saidinesh5 on Wednesday March 15, 2023 @01:23PM (#35171703)
I think they are using llama.cpp without any NPU/TPU patches. By default it only runs on CPU with support for various SIMD extensions.

https://github.com/ggerganov/llama.cpp

Re:LLaMa running at 5 tokens/second on a Pixel 6 by zodester on Wednesday March 15, 2023 @01:25PM (#35171725)
It uses the ARM NEON extensions to the instruction set for SIMD (as far as I understand).