Re:PowerInfer: Fast Large Language Model Serving with a Cons...
by girvo on Tuesday December 19, 2023 @06:22PM
(#38703357)
From their GitHub[0]:
> Evaluation shows that PowerInfer attains an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs (including OPT-175B) on a single NVIDIA RTX 4090 GPU, only 18% lower than that achieved by a top-tier server-grade A100 GPU. This significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy.
Impressive result and very exciting if it holds true! The hybridisation idea (preloading known-"hot" neurons to the GPU and leaving "cold" ones on the CPU) is a neat one on the surface.
Re:PowerInfer: Fast Large Language Model Serving with a Cons...
by 3abiton on Tuesday December 19, 2023 @06:53PM
(#38703645)
They mention OPT-175B, but the currently only available models are Falcon-40B and Llama2 (up to 70B). Curious about the performance in human chatbot arena LLM leaderboard.
> Evaluation shows that PowerInfer attains an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs (including OPT-175B) on a single NVIDIA RTX 4090 GPU, only 18% lower than that achieved by a top-tier server-grade A100 GPU. This significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy.
Impressive result and very exciting if it holds true! The hybridisation idea (preloading known-"hot" neurons to the GPU and leaving "cold" ones on the CPU) is a neat one on the surface.
[0] https://github.com/SJTU-IPADS/PowerInfer