News for hackers. Stuff that matters.
Re:Qwen4-Max by esperent on Sunday November 15, 2026 @10:32PM (#50513693)
[delayed]
Re:Qwen4-Max by lcnPylGDnU4H9OF on Sunday November 15, 2026 @10:56PM (#50513829)
> If they are not testing Chinese models with less than max thinking, then we should only compare them to other models also at max thinking.

They compare different reasoning levels.

https://artificialanalysis.ai/models/deepseek-v4-1-flash-101...

Re:Qwen4-Max by Aurornis on Sunday November 15, 2026 @11:48PM (#50514150)
The Artificial Analysis charts show max effort by default, which isn't how most people use them.

Scroll down to the cost per task pareto front chart which includes more difficult levels by default: https://artificialanalysis.ai/models/qwen4-max#intelligence-...

Qwen4-Max is not in a good spot on that chart. Astra xhigh and Fable 5.2 medium had a similar cost per task with higher intelligence ratings.

Re:Qwen4-Max by SV_BubbleTime on Sunday November 15, 2026 @11:49PM (#50514158)
How “safe” is Qwen4? Will it immediately shut down on security research like the US frontier models?
Re:Qwen4-Max by lambda on Monday November 16, 2026 @01:13AM (#50514579)
Not sure about Qwen 4. I have a coworker who's been doing some pretty good reverse engineering with Kimi K3.1 recently.
Re:Qwen4-Max by esperent on Sunday November 15, 2026 @11:57PM (#50514211)
[delayed]
Re:Qwen4-Max by lambda on Monday November 16, 2026 @01:08AM (#50514554)
They don't always run the benchmarks at every reasoning level for every model; only a few prominent ones. They do it for the US frontier labs, plus a few of the Chinese labs, but not consistently. You can see that DeepSeek has a few reasoning levels tested, GLM 5.3 has a couple, as does Qwen 3.8 27B, but not other Qwen models, and a lot of the Chinese models they only run at max reasoning.

It's possible that they will run Qwen 4 Max at other reasoning levels in the future, or it's possible that they never will. Regardless, it's unlikely to be better at any reasoning level on a price/performance basis than the GPT 6.1 Sol/GPT 6 Astra pareto frontier. It's close enough to Claude Opus 5.5 line at lower reasoning levels that it could potentially cross that if it doesn't loose too much performance at lower reasoning levels. But it's far enough from the global pareto frontier that it's unlikely that it would cross that at any resoning level.

Re:Qwen4-Max by esperent on Monday November 16, 2026 @01:21AM (#50514627)
[delayed]
Re:Qwen4-Max by brainwad on Monday November 16, 2026 @02:17AM (#50514946)
No, it's only valid to compare to models at the same cost level, regardless if that's their max level or not. Fable having higher effort modes than Qwen doesn't mean that you can't compare Fable medium to Qwen max.
Re:Qwen4-Max by esperent on Monday November 16, 2026 @03:30AM (#50515335)
[delayed]
Re:Qwen4-Max by aenis on Monday November 16, 2026 @02:56AM (#50515163)
> You can see for example that the cost per task for 6.1 sol is about 4x higher at max compared to medium thinking level.

It does not (universally) hold for long running workloads -- which is what people are increasingly doing.

Leave a Sonnet on medium, it will complete the task, spend more time and possibly significantly more token money, vs. Opus 5.5 on xhigh.

Re:Qwen4-Max by behnamoh on Monday November 16, 2026 @01:28AM (#50514668)
> The Artificial Analysis charts show max effort by default, which isn't how most people use them.

I always use Max effort in both Opus and Fable. It's actually faster overall because tasks get done faster with fewer "corrections" from my end.

On GPT models, I'd sometimes dabble in xhigh/high but that's because OpenAI usage quota are too limited.

Re:Qwen4-Max by esperent on Monday November 16, 2026 @03:30AM (#50515339)
[delayed]
Re:Qwen4-Max by athrowaway3z on Monday November 16, 2026 @01:23AM (#50514638)
> If they are not testing Chinese models with less than max thinking, then we should only compare them to other models also at max thinking.

This is wrong.

To make a vacuum analogy, one brand's Low setting could be another's Medium setting.

To compare you either give them the same resources ( Watts or $ ) and see how much they can do, or you give them a task and then check how many resources they required to finish it.

The label on the knob is irrelevant.

Re:Qwen4-Max by dtech on Monday November 16, 2026 @02:11AM (#50514905)
This is a silly take. To continue the analogy vacuum A might need the power at 50% to vacuum something, B 100%. You cannot then test both at 100% and say "both finished the task, but A required twice the resources so it's worse", since you didn't check what resources it actually needed.
Re:Qwen4-Max by parineum on Monday November 16, 2026 @02:31AM (#50515026)
The resources are dollars and the output is tasks. Choose whatever reasoning makes it work best and compare
Re:Qwen4-Max by PunchTornado on Monday November 16, 2026 @03:13AM (#50515245)
Are the resources only dollars? We shouldn't care about energy consumption? Water? It is difficult to calculate but we shouldn't forget about it.
Re:Qwen4-Max by baq on Monday November 16, 2026 @03:26AM (#50515312)
The idea is that dollars price in everything. It’s usually incorrect, but it’s also usually correct enough; and still much better than paying for services with water and joules directly
Re:Qwen4-Max by cyanydeez on Monday November 16, 2026 @03:40AM (#50515403)
Yes. The idea than money is equal to everything is kinda why we are all going to create the torment nexus. As it will be the most efficient way to torment civilization without all that effort.
Re:Qwen4-Max by baq on Monday November 16, 2026 @03:46AM (#50515437)
Money helps allocate resources efficiently, that’s all there is to it. It’s about as good or bad as guns. If we didn’t have money, someone would invent it and in fact it was invented more than once in different cultures.
Re:Qwen4-Max by cyanydeez on Monday November 16, 2026 @10:25AM (#50518589)
I don't deny that; but it rewires some peoples brains to the point that everything is adjustable via cash, including people, including their emotions, etc.

It's very clear part of America's diesease is the idea that everything can equate to a dollar cost.

Re:Qwen4-Max by parineum on Monday November 16, 2026 @10:19AM (#50518517)
Energy and water both cost dollars.
Re:Qwen4-Max by throawayonthe on Monday November 16, 2026 @02:32AM (#50515028)
isn't that exactly the argument?
Re:Qwen4-Max by esperent on Monday November 16, 2026 @04:47AM (#50515791)
[delayed]
Re:Qwen4-Max by tough on Monday November 16, 2026 @08:17AM (#50517315)
sounds like context collapse
Re:Qwen4-Max by spider-mario on Monday November 16, 2026 @09:33AM (#50518007)
Well, sure, it would be better to evaluate all of them throughout the range. But it does not follow that we should further restrict the information we do have in this specific way:

> If they are not testing Chinese models with less than max thinking, then we should only compare them to other models also at max thinking.

Re:Qwen4-Max by paimapi on Monday November 16, 2026 @10:12AM (#50518442)
Usage? If most people predominantly default to Medium thinking level for usage then it would make sense to benchmark that first.

Another analogy for this are review sites overly focusing on reviewing all the SKUs of GPUs that cost $1k+ for an audience that, according to Steam's Hardware Survey, predominantly use midrange X060s-equivalent cards. Sure, it's fun to know whether or not you can enable PT on a 5090 but for most of us, covering 4070 TIs and below in-depth (meaning actually taking the card apart, seeing how good/bad the past jobs are, possibly buying multiples of the same SKUs to evaluate QC, etc) is much more useful because of the value prop.

Re:Qwen4-Max by spider-mario on Monday November 16, 2026 @10:39AM (#50518760)
> Usage? If most people predominantly default to Medium thinking level for usage then it would make sense to benchmark that first.

Sure, that’s not in question. The discussion here is not about what to measure (we agree that benchmarking only max is suboptimal) but how to compare if you have a fixed set of measurements, outside of your control, which includes only “max” for model A but a wider range from model B.

The person I was responding to was claiming that the sensible thing to do, then, was to truncate the model B measurements to “only max” as well.

Re:Qwen4-Max by aenis on Monday November 16, 2026 @02:19AM (#50514961)
> If they are not testing Chinese models with less than max thinking, then we should only compare them to other models also at max thinking.

Nah, there are interesting conclusions to be had in comparing various model/effort level combinations, esp. when it comes to cost and time per task.

I have my own private benchmark suite; it involves algorithmic problems, the need for good visual understanding, and the ability to run long workloads unattended (as the agents need to run a few iterations, self-evaluate the results, and hand over the assignments when they feel they have succeeded). Chinese models do poorly on such problems, and running them at anything less than max efforts is pointless; they can't even hand over the assignments graded against a mechanical set of tests they themselves can execute. Hand that same task over to, say, Sonnet medium and you have a weird outcome: excellent result, but way more time and tokens burned than, say, Opus 5.5 or Sol 6.1 running at high. I do those benchmarks just to be able to spot such patterns and then better choose the models for my real work - but i wouldn't have caught it if I wasnt running various model/effort combos.

Re:Qwen4-Max by croon on Monday November 16, 2026 @04:20AM (#50515622)
I assume those cost comparisons are on API token cost.

I'd love a comparison on for example monthly quota usage of all ~$20/month subscriptions.