News for hackers. Stuff that matters.
Re:Qwen4-Max by anon373839 on Sunday November 15, 2026 @10:07PM (#50513556)
AA has become pretty meaningless. They just keep adjusting the recipe to get a preferred ranking result, with no supporting evidence for their decisions.
Re:Qwen4-Max by gr_norm on Sunday November 15, 2026 @10:30PM (#50513677)
Yeah, I don't know why people bother with it anymore. It no longer reflects what you're likely to see in real-world usage, at least IME.
Re:Qwen4-Max by stratts on Sunday November 15, 2026 @10:31PM (#50513686)
One thing I find confusing about AA is that they show GPT 6.1 dominating the lower end of the Pareto line, with eg, GPT 6.1 Sol at medium supposedly costing far less per task than DeepSeek V4.1 Flash at high.

Yet every time I compare the two in something like Pi, Sol ends up being 5-10x the price for a simple query, something like 7-10c vs. less than 0.5c for DeepSeek. What's the deal?

Re:Qwen4-Max by blackoil on Sunday November 15, 2026 @10:37PM (#50513725)
Same for me coding tasks taking $20-50 are easily done by deepseek in $5-10
Re:Qwen4-Max by the_duke on Sunday November 15, 2026 @10:43PM (#50513760)
To be fair, a lot of the popular benchmarks have become pretty meaningless.

Models often get great scores despite being a lot worse than the frontier in practice.

Re:Qwen4-Max by anon373839 on Monday November 16, 2026 @12:09AM (#50514288)
This is true, but if you look at the change in weighting to specific benchmarks from one version of AA to the next, it looks really arbitrary.
Re:Qwen4-Max by stymaar on Monday November 16, 2026 @02:35AM (#50515043)
A benchmark is only relevant if it was released after the model it evaluates.
Re:Qwen4-Max by wasabi991011 on Monday November 16, 2026 @12:49AM (#50514467)
I thought they were an independent company, what motive would they have to prefer some ranking results over others?
Re:Qwen4-Max by stymaar on Monday November 16, 2026 @02:33AM (#50515038)
> AA has become pretty meaningless

AA has always been miningless, but people keep referring yo it anyway just because the synthetic view is too convenient (even if meaningless).

Re:Qwen4-Max by mysteriousbow on Monday November 16, 2026 @11:19AM (#50519294)
> test Anthropic and OpenAI in their native harnesses, while not doing the same for Qwen/Deepseek/etc.

they keep harnesses the same across models other than in the coding agent index (https://artificialanalysis.ai/agents/coding-agents), which uses native harnesses for all models too