News for hackers. Stuff that matters.
Re:21.2× faster than llama.cpp? plus 40% memory usage reduction by worstspotgain on Wednesday June 12, 2024 @06:34PM (#40663932)
The speed improvement is only for models that don't entirely fit in memory, i.e. memory-starved llama.cpp degenerates to ~20x slower.

However, this scheme does reduce memory usage by 40%, meaning it allows models that are 67% bigger. It's a quality improvement, not a performance one.

> For models that fit entirely within the memory, PowerInfer-2 can achieve approximately a 40% reduction in memory usage while maintaining inference speeds comparable to llama.cpp and MLC-LLM.

Re:21.2× faster than llama.cpp? plus 40% memory usage reduction by russianGuy83829 on Wednesday June 12, 2024 @06:32PM (#40663922)
It seems like this can’t run all models, and needs custom ones trained from scratch: “ We introduce two new models: TurboSparse-Mistral-7B and TurboSparse-Mixtral-47B. These models are sparsified versions of Mistral and Mixtral […]. Notbly, our models are trained with just 150B tokens within just 0.1M dollars”.

It remains to be seen how good these custom models are.

Re:21.2× faster than llama.cpp? plus 40% memory usage reduction by TOMDM on Wednesday June 12, 2024 @06:41PM (#40663986)
Paper for the sparcified mixtral models

https://arxiv.org/abs/2406.05955

Re:21.2× faster than llama.cpp? plus 40% memory usage reduction by cpldcpu on Wednesday June 12, 2024 @08:14PM (#40664578)
This seems very similar to the other paper.
Re:21.2× faster than llama.cpp? plus 40% memory usage reduction by helloericsf on Wednesday June 12, 2024 @06:43PM (#40664009)
Agreed. custom models could be a hit or a miss.
Re:21.2× faster than llama.cpp? plus 40% memory usage reduction by cpldcpu on Wednesday June 12, 2024 @09:17PM (#40664944)
It's just continued pretraining to "heal" the damage caused by switching the activation functions and enforcing sparsity.

Apparently they managed to recover original performance on standardized tests after continuing pretraining with the 150B tokens. There may be some more specialized knowledge lost that was not covered by their dataset.

Re:21.2× faster than llama.cpp? plus 40% memory usage reduction by helloericsf on Wednesday June 12, 2024 @05:58PM (#40663668)
Re:21.2× faster than llama.cpp? plus 40% memory usage reduction by talldayo on Wednesday June 12, 2024 @06:01PM (#40663690)
The video/litepaper they linked is also a great read: https://powerinfer.ai/v2/
Re:21.2× faster than llama.cpp? plus 40% memory usage reduction by helloericsf on Wednesday June 12, 2024 @06:11PM (#40663773)
Truly mind-blowing to see Mixtral 47B running on a smartphone.
Re:21.2× faster than llama.cpp? plus 40% memory usage reduction by westurner on Wednesday June 12, 2024 @06:27PM (#40663877)
> TL;DR PowerInfer is a CPU/GPU LLM inference engine leveraging activation locality for your device.
Re:21.2× faster than llama.cpp? plus 40% memory usage reduction by wmf on Wednesday June 12, 2024 @06:29PM (#40663900)
So is it running on the phone or not?
Re:21.2× faster than llama.cpp? plus 40% memory usage reduction by westurner on Wednesday June 12, 2024 @06:39PM (#40663978)
From the abstract https://arxiv.org/abs/2406.06282 :

> Notably, PowerInfer-2 is the first system to serve the TurboSparse-Mixtral-47B model with a generation rate of 11.68 tokens per second on a smartphone. For models that fit entirely within the memory, PowerInfer-2 can achieve approximately a 40% reduction in memory usage while maintaining inference speeds comparable to llama.cpp and MLC-LLM

What does this do with a 40 TOPS+ NPU/TPU?

Re:21.2× faster than llama.cpp? plus 40% memory usage reduction by x0n on Wednesday June 12, 2024 @06:27PM (#40663880)
The hyperbollocks marketingspeak in the summary paragraph put me off:

"The key insight of PowerInfer-2 is to utilize the heterogeneous computation, memory, and I/O resources in smartphones by decomposing traditional matrix computations into fine-grained neuron cluster computations. Specifically, PowerInfer-2 features a polymorphic neuron engine that adapts computational strategies for various stages of LLM inference. Additionally, it introduces segmented neuron caching and fine-grained neuron-cluster-level pipelining, which effectively minimize and conceal the overhead caused by I/O operations."

Ahem, what? Let's overload a biological construct "neuron" to imbue it with magical technopowers and then derive the rest of our BS from this. No sale.

Re:21.2× faster than llama.cpp? plus 40% memory usage reduction by wakawaka28 on Wednesday June 12, 2024 @07:20PM (#40664257)
"Neuron" is not out of place when you're talking about software that uses neural networks...
Re:21.2× faster than llama.cpp? plus 40% memory usage reduction by samatman on Wednesday June 12, 2024 @06:34PM (#40663935)
Processes can't have children either, since, being a bunch of arithmetic, they lack genitalia. You can't kill them, because they aren't alive. They can't even sleep(1), another biological process fast arithmetic has no use for.

When will the false advertising end?!?