News for hackers. Stuff that matters.
Re:Qwen4-Max by puppymaster on Monday November 16, 2026 @01:37AM (#50514716)
AA must be doing something right. Ever since they released the new benchmark, you see open weight China model fans complaining about it under every release thread. Probably because it’s hard to benchmaxxed against.

But admitting their model is weaker is a no go. So conspiracy theory here we go

Re:Qwen4-Max by dtech on Monday November 16, 2026 @02:16AM (#50514937)
For what it's worth, in our own metrics and benchmarks the intelligence is a pretty good match, but the cost per task is wildly out of whack. We see the open weight models like GLM-5.3-flash or Deepseek 4.1 Flash 1017 use a fraction of their tokens/cost of the benchmarks, which changes the cost-benefit equation a lot.
Re:Qwen4-Max by anon373839 on Monday November 16, 2026 @02:19AM (#50514962)
> Probably because it’s hard to benchmaxxed against.

It's mainly just a composite of existing benchmarks. It would be very easy to benchmax if, say, AA shared with a preferred lab which benchmarks it plans to upweight/downweight in the next version. Just speaking hypothetically, of course.

Re:Qwen4-Max by puppymaster on Monday November 16, 2026 @02:33AM (#50515034)
Hence the constant switching around and running the benchmark themselves. They have been pretty transparent about which benchmarks get added or removed with every release.
Re:Qwen4-Max by rvba on Monday November 16, 2026 @02:20AM (#50514968)
or maybe it's the other way round, open weight models are under siege while closed weight models are defended