News for hackers. Stuff that matters.
Re:Qwen4-Max by rbehrends on Sunday November 15, 2026 @10:45PM (#50513770)
While I doubt that Qwen 4 Max can seriously compete with frontier models, the AA index has become all but useless, due to the introduction of new benchmarks in the aggregate which are of questionable value as a measure of "intelligence" [1]. AA has seemingly also abolished updating the Coding Agents Index, which was an okay benchmark for coding ability, and older models are completely absent from the v5 version of the full index.

[1] Especially Terminal-Bench-Science 0.1 stands out. Example problems:

- https://hub.harborframework.com/tasks/terminal-bench-science...

- https://hub.harborframework.com/tasks/terminal-bench-science...

Probably very relevant to people working in these fields, but it is honestly not surprising if a lab with limited compute will not train for these technologies, and it says pretty much nothing about more common use cases. This is where you really need a breakdown of capabilities and not an increasingly meaningless aggregate. I also suspect that this benchmark measures knowledge retrieval more than raw reasoning ability.

Re:Qwen4-Max by izabera on Sunday November 15, 2026 @11:19PM (#50513980)
alibaba has "limited compute"?
Re:Qwen4-Max by rbehrends on Sunday November 15, 2026 @11:26PM (#50514034)
Compared to OpenAI and Anthropic, yes. There's this export ban on Nvidia chips, after all. And all the CPUs they have won't help much with model training.
Re:Qwen4-Max by mitthrowaway2 on Monday November 16, 2026 @01:26AM (#50514660)
Can't they rent GPU time in American data centers like everyone else does?
Re:Qwen4-Max by j2j8 on Monday November 16, 2026 @01:38AM (#50514720)
They do in huge Singaporean ones like Megaspeed.
Re:Qwen4-Max by mrheosuper on Monday November 16, 2026 @04:53AM (#50515827)
That and limited training data.
Re:Qwen4-Max by throwa356262 on Monday November 16, 2026 @08:15AM (#50517300)
I am surprised that this is a problem for China.

CCP can probably "volunteer" a couple of thousand to build a high quality national training data bank.

Re:Qwen4-Max by gardnr on Monday November 16, 2026 @12:27AM (#50514356)
Did AA take on funding or something?
Re:Qwen4-Max by alfiedotwtf on Monday November 16, 2026 @12:50AM (#50514473)
Not only that. I find it weird that some models like Laguna S 2.1 were never included.

Maybe somebody should archive the scores and see if they in the future ever backtrack particular models to favour others (like last week)