News for hackers. Stuff that matters.
Re:Qwen4-Max by paimapi on Monday November 16, 2026 @10:12AM (#50518442)
Usage? If most people predominantly default to Medium thinking level for usage then it would make sense to benchmark that first.

Another analogy for this are review sites overly focusing on reviewing all the SKUs of GPUs that cost $1k+ for an audience that, according to Steam's Hardware Survey, predominantly use midrange X060s-equivalent cards. Sure, it's fun to know whether or not you can enable PT on a 5090 but for most of us, covering 4070 TIs and below in-depth (meaning actually taking the card apart, seeing how good/bad the past jobs are, possibly buying multiples of the same SKUs to evaluate QC, etc) is much more useful because of the value prop.

Re:Qwen4-Max by spider-mario on Monday November 16, 2026 @10:39AM (#50518760)
> Usage? If most people predominantly default to Medium thinking level for usage then it would make sense to benchmark that first.

Sure, that’s not in question. The discussion here is not about what to measure (we agree that benchmarking only max is suboptimal) but how to compare if you have a fixed set of measurements, outside of your control, which includes only “max” for model A but a wider range from model B.

The person I was responding to was claiming that the sensible thing to do, then, was to truncate the model B measurements to “only max” as well.