Unsloth publishes KL divergence numbers which measures how much the quantised probability distribution changes compared to unquantised: https://unsloth.ai/docs/models/qwen3.8#quantization-analysis

It's a bit bare at the moment, I assume they are going to add further detail later, similar to their other releases.

KL divergence is nothing close to a replacement for benchmarks. As flawed as benchmarks are, KL divergence is a barely useful signal. The fact that Unsloth only just started publishing KL divergences shows how unserious the quantization space is.

Actually we do publish non KLD benchmarks - top-1% is better - for NVFP4 for eg we did MMLU Pro, GPQA, AIME 2025: https://unsloth.ai/docs/models/qwen3.6#nvfp4-benchmarks

Sometimes they're just slow and expensive, so we we KLD as a proxy measure and it's very high correlation (95%+)

You're very right about KL divergence. I spent a couple days playing with the Gemma 4 models. That's 10 separate models (varying weights, MoE, QAT or not, etc...) with identical tokenizers. I treated 31B at BF16 as the gold standard, feeding Wikipedia snippets, and anthropomorphizing a bit:

Gemma 4 31B: "Um, if I really said all of that, I guess I'd say this next"

Gemma 4 26B: "Dude, I would've said completely different stuff" (large divergence)

Gemma 4 12B: "Umm, there's zero chance I would've said some of this" (INFINITE divergence)

Gemma 4 E4B and E2B: "Derp derp, I'm happy to say almost anything" (lowest divergence)

For models which are chat trained, they simply would not recite Wikipedia, so the divergence is almost meaningless. I thought about capturing a realistic coding session and trying to use that as the corpus, but you need to preserve the turn-based tokens and such, so I moved on to other things.

> The fact that Unsloth only just started publishing KL divergences shows how unserious the quantization space is.

Just wanted to say that this is a very important point that I totally agree with. People are obsessed with KL divergence, but it is yet to be demonstrated to be a descent proxy for agentic coding benchmarks.

It's a cheap to evaluate proxy for totally broken or not, which is a good start.

It also has a lot of resolution and not a lot of noise. Better would be multi-turn benchmarks with tools but getting good precision and accuracy for that is hard and computationally expensive.

> proxy for totally broken or not

You can't really know that either.

That's not a replacement for benchmarks

The talk around KL divergence is oversold. People talk about it as if it’s not a benchmark, but at its core it is in the same neighborhood! You get a different KL divergence number on different “calibration datasets”, so in other words it’s data-dependent. It is NOT a universal guarantee about the fundamental divergence of a model.

When I tested various eval benchmarks on Qwen3.5/3.6 27B with Unsloth's quants, the scores usually dropped 0-5% between UD-Q6 and UD-Q3 depending on the eval.