But quantized local models also dip in evals.
Re:Claude Fable 5.2 and Mythos 5.2
by rybosworld on Monday November 09, 2026 @03:45PM
(#50431824)
It doesn't matter if it's local or not, I just suggested local so people could test themselves.
But no, evals and benchmarks do not detect quantization, until around 4 bit quantization.
https://developers.redhat.com/articles/2024/10/17/we-ran-ove...
https://pub.towardsai.net/your-model-passed-safety-evals-the...
They don't detect a quantized model until the quantization gets down to 4 bits or less.
Humans absolutely do detect quantization well before 4 bits...
Try it yourself if you want. Try a small local model, both quantized and not, and I would be very surprised if it isn't immediately obvious which is which.