I’m going to laugh so hard if we finally find out that quantization really does hurt models, and all the claims of 99.9999% at 4bit or lower precision really were full of it!

Of all the gaslighting I see in Ml/AI, the idea that quantization is almost free is up there. I don’t buy it for one second at all, no matter how many charts you try to show me implying that the logprobs are 99% the same. Sure, maybe until you go to 10K tokens context window!