News for hackers. Stuff that matters.
Re:Qwen4-Max by athrowaway3z on Monday November 16, 2026 @01:23AM (#50514638)
> If they are not testing Chinese models with less than max thinking, then we should only compare them to other models also at max thinking.

This is wrong.

To make a vacuum analogy, one brand's Low setting could be another's Medium setting.

To compare you either give them the same resources ( Watts or $ ) and see how much they can do, or you give them a task and then check how many resources they required to finish it.

The label on the knob is irrelevant.

Re:Qwen4-Max by dtech on Monday November 16, 2026 @02:11AM (#50514905)
This is a silly take. To continue the analogy vacuum A might need the power at 50% to vacuum something, B 100%. You cannot then test both at 100% and say "both finished the task, but A required twice the resources so it's worse", since you didn't check what resources it actually needed.
Re:Qwen4-Max by parineum on Monday November 16, 2026 @02:31AM (#50515026)
The resources are dollars and the output is tasks. Choose whatever reasoning makes it work best and compare
Re:Qwen4-Max by PunchTornado on Monday November 16, 2026 @03:13AM (#50515245)
Are the resources only dollars? We shouldn't care about energy consumption? Water? It is difficult to calculate but we shouldn't forget about it.
Re:Qwen4-Max by baq on Monday November 16, 2026 @03:26AM (#50515312)
The idea is that dollars price in everything. It’s usually incorrect, but it’s also usually correct enough; and still much better than paying for services with water and joules directly
Re:Qwen4-Max by cyanydeez on Monday November 16, 2026 @03:40AM (#50515403)
Yes. The idea than money is equal to everything is kinda why we are all going to create the torment nexus. As it will be the most efficient way to torment civilization without all that effort.
Re:Qwen4-Max by baq on Monday November 16, 2026 @03:46AM (#50515437)
Money helps allocate resources efficiently, that’s all there is to it. It’s about as good or bad as guns. If we didn’t have money, someone would invent it and in fact it was invented more than once in different cultures.
Re:Qwen4-Max by cyanydeez on Monday November 16, 2026 @10:25AM (#50518589)
I don't deny that; but it rewires some peoples brains to the point that everything is adjustable via cash, including people, including their emotions, etc.

It's very clear part of America's diesease is the idea that everything can equate to a dollar cost.

Re:Qwen4-Max by parineum on Monday November 16, 2026 @10:19AM (#50518517)
Energy and water both cost dollars.
Re:Qwen4-Max by throawayonthe on Monday November 16, 2026 @02:32AM (#50515028)
isn't that exactly the argument?
Re:Qwen4-Max by esperent on Monday November 16, 2026 @04:47AM (#50515791)
[delayed]
Re:Qwen4-Max by tough on Monday November 16, 2026 @08:17AM (#50517315)
sounds like context collapse
Re:Qwen4-Max by spider-mario on Monday November 16, 2026 @09:33AM (#50518007)
Well, sure, it would be better to evaluate all of them throughout the range. But it does not follow that we should further restrict the information we do have in this specific way:

> If they are not testing Chinese models with less than max thinking, then we should only compare them to other models also at max thinking.

Re:Qwen4-Max by paimapi on Monday November 16, 2026 @10:12AM (#50518442)
Usage? If most people predominantly default to Medium thinking level for usage then it would make sense to benchmark that first.

Another analogy for this are review sites overly focusing on reviewing all the SKUs of GPUs that cost $1k+ for an audience that, according to Steam's Hardware Survey, predominantly use midrange X060s-equivalent cards. Sure, it's fun to know whether or not you can enable PT on a 5090 but for most of us, covering 4070 TIs and below in-depth (meaning actually taking the card apart, seeing how good/bad the past jobs are, possibly buying multiples of the same SKUs to evaluate QC, etc) is much more useful because of the value prop.

Re:Qwen4-Max by spider-mario on Monday November 16, 2026 @10:39AM (#50518760)
> Usage? If most people predominantly default to Medium thinking level for usage then it would make sense to benchmark that first.

Sure, that’s not in question. The discussion here is not about what to measure (we agree that benchmarking only max is suboptimal) but how to compare if you have a fixed set of measurements, outside of your control, which includes only “max” for model A but a wider range from model B.

The person I was responding to was claiming that the sensible thing to do, then, was to truncate the model B measurements to “only max” as well.