They compare different reasoning levels.
https://artificialanalysis.ai/models/deepseek-v4-1-flash-101...
Scroll down to the cost per task pareto front chart which includes more difficult levels by default: https://artificialanalysis.ai/models/qwen4-max#intelligence-...
Qwen4-Max is not in a good spot on that chart. Astra xhigh and Fable 5.2 medium had a similar cost per task with higher intelligence ratings.
It's possible that they will run Qwen 4 Max at other reasoning levels in the future, or it's possible that they never will. Regardless, it's unlikely to be better at any reasoning level on a price/performance basis than the GPT 6.1 Sol/GPT 6 Astra pareto frontier. It's close enough to Claude Opus 5.5 line at lower reasoning levels that it could potentially cross that if it doesn't loose too much performance at lower reasoning levels. But it's far enough from the global pareto frontier that it's unlikely that it would cross that at any resoning level.
It does not (universally) hold for long running workloads -- which is what people are increasingly doing.
Leave a Sonnet on medium, it will complete the task, spend more time and possibly significantly more token money, vs. Opus 5.5 on xhigh.
I always use Max effort in both Opus and Fable. It's actually faster overall because tasks get done faster with fewer "corrections" from my end.
On GPT models, I'd sometimes dabble in xhigh/high but that's because OpenAI usage quota are too limited.
This is wrong.
To make a vacuum analogy, one brand's Low setting could be another's Medium setting.
To compare you either give them the same resources ( Watts or $ ) and see how much they can do, or you give them a task and then check how many resources they required to finish it.
The label on the knob is irrelevant.
It's very clear part of America's diesease is the idea that everything can equate to a dollar cost.
> If they are not testing Chinese models with less than max thinking, then we should only compare them to other models also at max thinking.
Another analogy for this are review sites overly focusing on reviewing all the SKUs of GPUs that cost $1k+ for an audience that, according to Steam's Hardware Survey, predominantly use midrange X060s-equivalent cards. Sure, it's fun to know whether or not you can enable PT on a 5090 but for most of us, covering 4070 TIs and below in-depth (meaning actually taking the card apart, seeing how good/bad the past jobs are, possibly buying multiples of the same SKUs to evaluate QC, etc) is much more useful because of the value prop.
Sure, that’s not in question. The discussion here is not about what to measure (we agree that benchmarking only max is suboptimal) but how to compare if you have a fixed set of measurements, outside of your control, which includes only “max” for model A but a wider range from model B.
The person I was responding to was claiming that the sensible thing to do, then, was to truncate the model B measurements to “only max” as well.
Nah, there are interesting conclusions to be had in comparing various model/effort level combinations, esp. when it comes to cost and time per task.
I have my own private benchmark suite; it involves algorithmic problems, the need for good visual understanding, and the ability to run long workloads unattended (as the agents need to run a few iterations, self-evaluate the results, and hand over the assignments when they feel they have succeeded). Chinese models do poorly on such problems, and running them at anything less than max efforts is pointless; they can't even hand over the assignments graded against a mechanical set of tests they themselves can execute. Hand that same task over to, say, Sonnet medium and you have a weird outcome: excellent result, but way more time and tokens burned than, say, Opus 5.5 or Sol 6.1 running at high. I do those benchmarks just to be able to spot such patterns and then better choose the models for my real work - but i wouldn't have caught it if I wasnt running various model/effort combos.