News for hackers. Stuff that matters.
Re:21.2× faster than llama.cpp? plus 40% memory usage reduction by talldayo on Wednesday June 12, 2024 @06:01PM (#40663690)
The video/litepaper they linked is also a great read: https://powerinfer.ai/v2/
Re:21.2× faster than llama.cpp? plus 40% memory usage reduction by helloericsf on Wednesday June 12, 2024 @06:11PM (#40663773)
Truly mind-blowing to see Mixtral 47B running on a smartphone.
Re:21.2× faster than llama.cpp? plus 40% memory usage reduction by westurner on Wednesday June 12, 2024 @06:27PM (#40663877)
> TL;DR PowerInfer is a CPU/GPU LLM inference engine leveraging activation locality for your device.
Re:21.2× faster than llama.cpp? plus 40% memory usage reduction by wmf on Wednesday June 12, 2024 @06:29PM (#40663900)
So is it running on the phone or not?
Re:21.2× faster than llama.cpp? plus 40% memory usage reduction by westurner on Wednesday June 12, 2024 @06:39PM (#40663978)
From the abstract https://arxiv.org/abs/2406.06282 :

> Notably, PowerInfer-2 is the first system to serve the TurboSparse-Mixtral-47B model with a generation rate of 11.68 tokens per second on a smartphone. For models that fit entirely within the memory, PowerInfer-2 can achieve approximately a 40% reduction in memory usage while maintaining inference speeds comparable to llama.cpp and MLC-LLM

What does this do with a 40 TOPS+ NPU/TPU?