maxul

PowerInfer is a high-speed Large Language Model (LLM) inference engine on a personal computer (PC) equipped with a single consumer-grade GPU. The key underlying the design of PowerInfer is exploiting the high locality inherent in LLM inference, characterized by a power-law distribution in neuron activation.

PowerInfer attains an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs (including OPT-175B) on a single NVIDIA RTX 4090 GPU, only 18% lower than that achieved by a top-tier server-grade A100 GPU. This significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy.

Debugging a blackbox is challenging when a developer lacks access to the bug report, which contains sensitive user data. This is not uncommon, most commercial software, such as Windows and Office, generate anonymized bug reports from users' personal desktops, and they promise that user privacy will not be compromised.

However, in the realm of databases, blackbox debugging becomes significantly more difficult. The basic idea of this project is twofold: (i) using operators that transform a database into an encrypted version, enabling developers to examine its running states, and (ii) replacing user data with equivalent data classes (this concept exists in software engineering testing, where two inputs produce the same coverage and trigger the same bug).

An intriguing aspect of this work is its compilation of numerous common actions taken by database administrators (DBAs) and translating them into actionable insights, potentially opening doors for further research. One exciting possibility is to explore automatic local debugging using these insights, which eliminates the need for human involvement and addresses concerns about data breaches.