> Beats Opus 4.7 Max

I'm a huge open model fan, and have used them since forever, even have daily drivers for on-prem dev, but no. They do not beat opus on real-world usage.

Qwen models are impressively good for what they are, are "good enough" for plenty tasks, can be ran locally on decently priced hardware, and so on. They certainly have their uses, and the field in general has advanced faster than my early expectations. But to compare a 27B model to SotA behemoths from a few months ago is doing everyone a disservice, especially people who pick it up, try to use them just like API models, and leave disappointed and confused. Number goes up on a benchmark isn't it.

> They do not beat opus on real-world usage

We have an internal eval that measures performance on tasks for a handful of embedded systems repos for our mmWave radios (mostly Rust, some C for microcontroller stuff). Qwen3.6-27B scores only 4% lower for pass@1, n=250 compared to Opus-4.8.

For the labeled dataset, the average PR size they're being measured against is around 1.5k SLOC.

This is very much "real-world usage" for us. The sort of change sets that come in daily/weekly and are solving non-trivial issues in the respective codebases.

As is usually the case, the most broad claims from both the labs and from the consequent pushback are talking past each other.

Let us know when you have Qwen vs Qwen comparison stats. As long as there's not a regression, that'd be awesome.

4% is within the margin of error anyways for pass@1, so I think pass@k > 1 is gonna be the better indicator of any movement (still need to calibrate the optimal k to re-test). 10 seems too tolerant even though that tends to be the next tranche I reach for.

Depends on where you sit on the binomial curve. At p=0.04 for n=250 4% points would not be within margin of error.

Right I take your point, but that's not the comparison in this case though. These are paired results on the same 250 tasks, so whether a 4 point gap is meaningful depends on the task-level deltas, not just plugging p=0.04 into the binomial SE.

How much does it score though? 0% would be 4% less if Opus was at 4%. Unless you mean relative fraction not percentage points - but people usually mean percentage points in such situations.

0% is not 4% less than 4%, that would be 3.84%.

0% is 4 percentage points (pp) less than 4%.

>> We have an internal eval that measures performance on tasks for a handful of embedded systems repos for our mmWave radios

Okay but the parent said real-world usage, presumably meaning coding tasks.

We have a whole bunch of complex evals that Haiku 4.5 passes. That doesn't mean it is a good model for coding.

Yes, these are coding tasks in the embedded systems domain (I mentioned Rust and C).

They literally stated in their first sentence that it was coding tasks.

> ...but no. They do not beat opus on real-world usage.

I agree, but then we just need meaningful benchmarks that clearly show that! Otherwise it's hand waving about something that should be put on paper in quantifiable terms.

If you are working in a company and using language models, it is a very good idea to hold a bunch of evals you can trust and use to validate new models. Calibrate every once in a while with prod data. We have our own and the only numbers on quality and cost I trust come from this setup.

A wise man once said, "not everything that counts can be counted, and not everything that can be counted counts".

That’s well and good, but how are we supposed to evaluate the accuracy of random HN comments without anything resembling somewhat objective metrics? People say all manner of things, and usually it’s contradictory. What heuristic do you propose?

Only useful benchmarks are those you (in particular) don't have access to.

The only useful benchmarks are those you've created for your specific workflow. Only then can you assess whether a given model is better or worse for what you are using it for.

There are tools like promptfoo designed for this.

> but then we just need meaningful benchmarks that clearly show that!

That's the rub. AI benchmarks are IMO, by and large totally unreliable. We think of them as similar to traditional benchmarks of deterministic processes where the number of variables is low. But they're anything but that. Non-deterministic processes with an astounding number of variables and fuzzy acceptance criteria.

It leads to results like these, where if you take it at face value, the only conclusion you can draw is "wow Anthropic must be stupid if Opus takes 1T parameters to do what Qwen can do in 27B."

In the end, the only benchmark that matters is your own.

There is 0 shot you can make that claim about this model you have not used or downloaded yet

How can you say this when you haven't even tried it yet? Is it just hypothetical vibes?

"Benchmark is stupid" and "model beats model on benchmark" are two different things, though. The second one is objectively true regardless of your views on the first one, right? To expect everyone to share your opinion that benchmarks are stupid is pretty weird, and just saying "no" to an objective truth is the definition of delusion.

If a benchmark is a measure of nothing useful, then model beats model is an objectively useless fact

But that's the problem with that logic. The author subjectively claims benchmarks are stupid (a debatable opinion) and treats that as inherently true while making the second claim (hence, you shouldn't say model A beat model B).

I can say I find the law stupid, so no one should say person A beat person B in court. But I did not prove the law is stupid; I merely thought it subjectively and demanded others to follow the second part because I believe the first part is true.

Saying that "if the law is useless, court cases are useless" is objectively true and cannot be argued with. But you still need to prove why the law is useless, not why you think it is and even then if people disagree and use the law as a reference, then it's not objectively useless and court cases are not useless as well.

Yep. These small models are actually worse than GPT 3.5 at some tasks (like recalling facts). You can definitely make models smarter at specific tasks (like tool calling, coding) but you can't compress the entire human knowledge into a 30GB file. It's just not enough bits.

But why would you use a model to store factual knowledge, that is stupid. We want intelligence, not a database.

Models cant make any decisions if they have 0 idea that the feature exist in this language or in some general fact.

For example you would tell a model hey, become an expert in this language for me, search it online, it would still need to learn it and download the data to it's context and then increasing the memory usage, there's no way around it.

Implicit knowledge is different from remembering every church that exists in the state of NSW Australia

that is why we enable web search for the agent. the memory can come from the internet.

deepseek-v4-flash needs web search to return true facts.

Yeah but results are worse, see https://news.ycombinator.com/item?id=49301574

That's a good sign.

Ideal local model would not know stuff like who Britney Spears is, best to leave precious weights for something useful.

Of course the line is very blurry but I'd be perfectly happy with local model that doesn't know anything about history, geopolitics, art or even biology etc. just coding, operating systems etc.

At one point there were specific -Coder release of Qwen e.g. 2.5 but they dropped that, still wondering how much better 3.6-Coder or 3.8-Coder would be when they ignore everything else

In my experience, the facts that are compressed away in small models are ones you don’t need them to memorize. They need familiarity with the essential concepts in a field, so that they will have better comprehension of material put into the context (and make better retrieval decisions). They don’t need to know very particular details which are a hallucination risk and should be derived from the context instead.

Is there any advantage to using the model from Unsloth compared with https://huggingface.co/Qwen/Qwen3.8-27B-FP8 ?

Depends on what software/hardware you'll run it. GGUFs from Unsloth can run on pretty much every single potato; full weights need beefy gpus

We also made NVFP4 ones if that helps! https://huggingface.co/unsloth/Qwen3.8-27B-NVFP4

This is the version we'll be testing on our rtx 6000 today! Thank you

Why not just run FP8 on vLLM with that much vRAM? It's plenty fast.

For high concurrency, using the blackwell's native native W4A4 MLP compute path, nvfp4 is something like a 1.2-1.5x performance increase over FP8. We're doing data enrichment (so, tasks completed successfully + tokens/second) so the performance bump shows up in the tasks/month number.

I am just now getting the benchmarks running against 3.8 27b but I expect similar results from benching 3.6 27b at the same quant.

I see. Did you see any intelligence degradation between FP8 and NVFP4 for 3.6 27B? You're using vLLM, right?

I didn't have time to run this as well, but we're getting a 99% agent completion rate across all tasks and 98% task decision that matches the human selected option(s) on Qwen 3.8 27B @ NVFP4. There may be a difference between FP8 and NVFP4 but it's inconsequential for our data enrichment purposes.

Unsloth one is gguf for llama.cpp (and some other on-device engines).

So advantage is not having to produce your own quantisation / gguf from .safetensors you've linked.

Run the unsloth if you are using llama.cpp (GGUF)

Run the one you linked if you are running vllm (safetensors)

Unsloth usually also fixes the models when they bork something, which always happens. For Gemma for example the tool calling wasn't working for the longest time.

That wasn't our problem right? Gemma officially updated tool calling which we adopted

if you have the VRAM, use offical release. quantized model lose focus after long context and can do damages or thinking loop

Considering the clusterfuck that is opus 5 or even fable, if Qwen 27B is trully better than Opus 4.7 Max, I will rejoice.

If it's as good as Sonnet 4.6 for most things I'd be happy.

Same

Yeah Opus 5 is almost unusable as a daily driver without making me go insane from excessive claude babble.

I wish each quant was benchmarked on the same tests as the original network so we could compare their performance

Unsloth publishes KL divergence numbers which measures how much the quantised probability distribution changes compared to unquantised: https://unsloth.ai/docs/models/qwen3.8#quantization-analysis

It's a bit bare at the moment, I assume they are going to add further detail later, similar to their other releases.

KL divergence is nothing close to a replacement for benchmarks. As flawed as benchmarks are, KL divergence is a barely useful signal. The fact that Unsloth only just started publishing KL divergences shows how unserious the quantization space is.

Actually we do publish non KLD benchmarks - top-1% is better - for NVFP4 for eg we did MMLU Pro, GPQA, AIME 2025: https://unsloth.ai/docs/models/qwen3.6#nvfp4-benchmarks

Sometimes they're just slow and expensive, so we we KLD as a proxy measure and it's very high correlation (95%+)

You're very right about KL divergence. I spent a couple days playing with the Gemma 4 models. That's 10 separate models (varying weights, MoE, QAT or not, etc...) with identical tokenizers. I treated 31B at BF16 as the gold standard, feeding Wikipedia snippets, and anthropomorphizing a bit:

Gemma 4 31B: "Um, if I really said all of that, I guess I'd say this next"

Gemma 4 26B: "Dude, I would've said completely different stuff" (large divergence)

Gemma 4 12B: "Umm, there's zero chance I would've said some of this" (INFINITE divergence)

Gemma 4 E4B and E2B: "Derp derp, I'm happy to say almost anything" (lowest divergence)

For models which are chat trained, they simply would not recite Wikipedia, so the divergence is almost meaningless. I thought about capturing a realistic coding session and trying to use that as the corpus, but you need to preserve the turn-based tokens and such, so I moved on to other things.

> The fact that Unsloth only just started publishing KL divergences shows how unserious the quantization space is.

Just wanted to say that this is a very important point that I totally agree with. People are obsessed with KL divergence, but it is yet to be demonstrated to be a descent proxy for agentic coding benchmarks.

It's a cheap to evaluate proxy for totally broken or not, which is a good start.

It also has a lot of resolution and not a lot of noise. Better would be multi-turn benchmarks with tools but getting good precision and accuracy for that is hard and computationally expensive.

> proxy for totally broken or not

You can't really know that either.

That's not a replacement for benchmarks

The talk around KL divergence is oversold. People talk about it as if it’s not a benchmark, but at its core it is in the same neighborhood! You get a different KL divergence number on different “calibration datasets”, so in other words it’s data-dependent. It is NOT a universal guarantee about the fundamental divergence of a model.

When I tested various eval benchmarks on Qwen3.5/3.6 27B with Unsloth's quants, the scores usually dropped 0-5% between UD-Q6 and UD-Q3 depending on the eval.

That's crazy, considering the massive size difference. But the small Qwen models are known for punching above their weight.

Good morning Dario!

That kind of result makes me suspicious of benchmaxxing. Qwen 27B is 100x smaller than Opus 4.7. Is it really 100x more parameter-efficient? Two orders of magnitude is hard to believe. I don't have the hardware to run a 27B, but I'm curious what real world use is like. Maybe I'll have to buy some usage on a cloud provider to run my own tests, but this seems fishy to me.

Qwen small models are heavily coding focused, whereas Opus is everything to everybody (even if code is their bread and butter). The downside is they'll frequently hallucinate world knowledge so they need to be RL'd to double check their knowledge against sources and verify facts/library names/etc.

It's very agentic coding focused; and I'd say a good executor but certainly not Opus in scale; overall knowledge; long-horizon work and recovery; etc.

e.g. If you try to chat to it about something philosophical for example, or maybe a debate / creative writing, then you'll very quickly see how it is still a much smaller model at the end of the day.

Still, it's such a relatively accessible model to run, and I find a big part of leveraging smaller models is to give it well-scoped tasks; not too high level or ambitious ones. Very impressive for its size and the ability to run locally :)

Interesting! I'll definitely give that a shot then.

In the local LLM communities there is a lot of respect for the Qwen models, but everyone comes to acknowledge that they do a lot of benchmaxxing after using them. Even at full precision they're never as good as models with similar benchmarks.

Oh, goody goody goody! Can't wait for a Q4M with *MTP* support. Does anyone know if there are plans for it?

I am currently using Qwen 3.6 on RTX 3090 and I have to admit that without MTP it would be too slow to be acceptable for me (30-35 tok/sec without MTP, 60-70 with MTP). Without MTP I would just use OpenRouter and rather pay for speed despite having a capable local setup.

It comes with MTP already, they just don't mention it in the filename as it's now the default.

Oh, thank you for telling me this. Trying it out now.