Excuse my ignorance, but what is a “dense” model? Why is comparing GPT-OSS 20B against 8B dense models “fair” but not when comparing it against 32B models?

So gpt-oss-20b is a sparse MoE model

Which means it has ~3b parameters active per token.

Qwen3-32b has 32b params active per token

An MoE model will not activate most of the model for any given query, so there is zero compute happening across an increasing part of the model.

Dense model means 32B parameters => 32B get used in calculation for every token. Every calculation takes time, and assuming similar latent space size (which they all have). For example Qwen-32B

MoE model has for example 80B parameters, but only 3B get used in calculation for any given token. For example Qwen3-Next 80B A3B

Performance comparison:

Qwen-32B => 56 tok/sec, 32 GB of VRAM

Qwen3-Next 80B A3B => 167 tok/sec, 85 GB of VRAM

So despite being close to 3x "bigger", Qwen3-Next is more than 3 times faster with the same compute capacity. There's a but though. But because what gets activated from one token to the next is a different subset of the model, it is still critical to have all 80B parameters loaded into memory.

So MoE performs much better with less compute, at the cost of more memory. It also performs better on benchmarks, it is a better model.

Similar techniques have long been used in ML to great success, rather than trying to create one brilliant model, create many that each have pros and cons, and then train a second model to figure out the best model for the task in front of you.

There's other factors, the big one being attention. That's why non-attention models, like MAMBA, will wipe the floor in terms of performance per flop (a compute unit), with anything else. When it comes to intelligence however ...

You don't mention that MoE models horrible damage logprobs (by definition) and basically need a whole new theory of LLM sampling written for them.

Dense models are better for a reason, and the idea that "everyone is doing MoE now and dense models are dead" is total bunk nonsense.

You can quantize dense models, and 4 bit quantized Qwen 32B is still better than full precision GPT-OSS. Luckily Unsloth even gives you tools to go down to 1.58bits!

Qwen3 Next and Qwen3-30b-a3b are pretty decent proxies for GPT-OSS 120B and 20B respectively (and in fact are both MoEs with 3B active rather than 8B active parameters), and they lap GPT-OSS pretty hard in this specific benchmark, getting to #17 and #33 respectively. That being said, it's hard to take benchmarks beyond a grain of salt because real world tasks that I try to use these models for always have a lot more variation than the benchmarks illustrate. I do view GPT-OSS as a pretty good alternative to the Qwen models in some cases but there are tradeoffs -- while I see better reasoning from GPT-OSS sometimes, the prompt adherence and overall flexibility of the Qwen models makes them a lot better IMO as general purpose local open weight models.

Those models got released later than GPT-OSS. It’s like saying the Android phone released 6 months after the iPhone is faster. Maybe, but it also had 6 extra months of development.