Re:21.2× faster than llama.cpp? plus 40% memory usage reduction
by westurner on Wednesday June 12, 2024 @06:27PM
(#40663877)
> TL;DR PowerInfer is a CPU/GPU LLM inference engine leveraging activation locality for your device.
[ Parent
| on HN ]
Re:21.2× faster than llama.cpp? plus 40% memory usage reduction
by wmf on Wednesday June 12, 2024 @06:29PM
(#40663900)
So is it running on the phone or not?
Re:21.2× faster than llama.cpp? plus 40% memory usage reduction
by westurner on Wednesday June 12, 2024 @06:39PM
(#40663978)
From the abstract https://arxiv.org/abs/2406.06282 :
> Notably, PowerInfer-2 is the first system to serve the TurboSparse-Mixtral-47B model with a generation rate of 11.68 tokens per second on a smartphone. For models that fit entirely within the memory, PowerInfer-2 can achieve approximately a 40% reduction in memory usage while maintaining inference speeds comparable to llama.cpp and MLC-LLM
What does this do with a 40 TOPS+ NPU/TPU?