eldar_ciki
-> Long story short: when models are carefully quantized, everything looks good. The lowest accuracy recovery we found was 96% relative to unquantized baseline, and this happened only for 8B model at weight-only INT4 quantization mostly because the unquantized baseline model has close to random prediction accuracy on a couple of Leaderboard v2 benchmarks. As one could imagine, "recovering" random accuracy is a bit noisy.
Show HN: Panza: A personal email assistant, trained and running on-device
https://github.com/IST-DASLab/PanzaMail
Tired of crafting well-polished emails and wish you had an assistant to take over the hard work while mimicking your writing style? Introducing Panza, a personalized LLM email assistant that runs entirely on your device! Choose between Llama-3 or Mistral, tailor it to your unique style, and let it write the emails for you. Take a look at our demo and give it a try on your emails at:...