News for hackers. Stuff that matters.

eldar_ciki

Re:We Ran Over Half a Million Evaluations on Quantized LLMs on Friday October 18, 2024 @06:51AM (#41878165)
The ML community has recently questioned whether quantized LLMs can genuinely compete with their full-precision counterparts. To address this, we conducted over half a million evaluations on quantized Llama-3.1-{8B, 70B, 405B}-Instruct models across FP8, INT8, and INT4 quantization schemes. We looked at various benchmarks, from open-ended challenges like Arena-Hard, to rigorous academic benchmarks such as MMLU, MMLU-Pro, Big Bench Hard, ARC-Challenge, IFEval, GPQA (and others from OpenLLM Leaderboard v1 and v2), and coding tests like HumanEval and HumanEval+.

-> Long story short: when models are carefully quantized, everything looks good. The lowest accuracy recovery we found was 96% relative to unquantized baseline, and this happened only for 8B model at weight-only INT4 quantization mostly because the unquantized baseline model has close to random prediction accuracy on a couple of Leaderboard v2 benchmarks. As one could imagine, "recovering" random accuracy is a bit noisy.

you might want to open an issue in our git repo, and if there is interest we would be more than happy to add a solution for Apple silicon

Show HN: Panza: A personal email assistant, trained and running on-device

https://github.com/IST-DASLab/PanzaMail

Tired of crafting well-polished emails and wish you had an assistant to take over the hard work while mimicking your writing style? Introducing Panza, a personalized LLM email assistant that runs entirely on your device! Choose between Llama-3 or Mistral, tailor it to your unique style, and let it write the emails for you. Take a look at our demo and give it a try on your emails at:...

( Read More... | 4 comments | 2 points | on HN )