From their GitHub[0]:

> Evaluation shows that PowerInfer attains an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs (including OPT-175B) on a single NVIDIA RTX 4090 GPU, only 18% lower than that achieved by a top-tier server-grade A100 GPU. This significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy.

Impressive result and very exciting if it holds true! The hybridisation idea (preloading known-"hot" neurons to the GPU and leaving "cold" ones on the CPU) is a neat one on the surface.

[0] https://github.com/SJTU-IPADS/PowerInfer

They mention OPT-175B, but the currently only available models are Falcon-40B and Llama2 (up to 70B). Curious about the performance in human chatbot arena LLM leaderboard.

Really clever trick. GPU/CPU splits are currently painfully slow so this may just make them more bearable.

> PowerInfer’s source code is publicly available at https://github.com/SJTU-IPADS/PowerInfer

---

Just curious - PowerInfer seems to market itself by running very large models (40B, 70B) on something like a 4090. If I have, say, a 3060 12GB, and I want to run something like a 7B or 13B, can I expect the same speedup of around 10x? Or does this only help that much for models that wouldn't already fit in VRAM?

Probably, the idea that the GPU handles the "hot neurons" means that if your GPU is strong enough to handle _enough_ hot neurons, you should expect speedups.

The linked paper also includes performance of a second PC configuration. The 'high' configuration is a 4090 (24GB) with a 24-core cpu (iirc), 192 GB ram, pci 4. The 'low' configuration is a 2080ti (11GB) with an 8-core cpu, 64 GB ram, pci 3. The low configuration still averaged a 5x speedup over llama.cpp, so yeah, this is exciting. They noted that for particularly large models (60B+) the hot neurons don't all fit into the gpu's vram and performance falls off. Similarly, for small models with small context sizes enough fits into the gpu to begin with that the performance gains are less pronounced. So there's going to be a sweet spot with regard to the combination of model and context size for a particular configuration, but yes, this still gives a huge speedup compared to llama.cpp.

Image of the PC-low configuration's results: https://i.imgur.com/X5reGkd.png showing speedups ranging from 2x-13x.

> 24-core cpu

Pretty sure it's an 8-core CPU, or at least that's how they're using it. Their demo videos show only 8 threads in use, probably because the "efficiency cores" would negate some of the performance win.

That's an impressive result. LLMs should soon be much cheaper to run.

This is basically fork of llama.cpp. I created a PR to see the diff and added my comments on it: https://github.com/ggerganov/llama.cpp/pull/4543

One thing that caught my interest is this line from their readme:

> PowerInfer exploits such an insight to design a GPU-CPU hybrid inference engine: hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons are computed on the CPU, thus significantly reducing GPU memory demands and CPU-GPU data transfers.

Apple's Metal/M3 is perfect for this because CPU and GPU share memory. No need to do any data transfers. Checkout mlx from apple: https://github.com/ml-explore/mlx