News for hackers. Stuff that matters.
Re:Qwen4-Max by augment_me on Monday November 16, 2026 @12:08AM (#50514272)
Artificial Analysis is a VC-backed firm and it's unclear what the incentives are there.

They have altered their benchmark multiple times in the last 4 weeks (v5 in https://artificialanalysis.ai/articles/artificial-analysis-i...) and the last 5.0.1 change https://artificialanalysis.ai/methodology/intelligence-bench... pushed up Anthropic's models a bit, you can see that v5.0.0 set on the first link Opus 5.5 has a lower score than in v5.0.1, and v5.0.1 was released just before Fable dropped.

Some part of me believes that the benchmark mix was put together to favor Anthropic's models.

Re:Qwen4-Max by puppymaster on Monday November 16, 2026 @01:37AM (#50514716)
AA must be doing something right. Ever since they released the new benchmark, you see open weight China model fans complaining about it under every release thread. Probably because it’s hard to benchmaxxed against.

But admitting their model is weaker is a no go. So conspiracy theory here we go

Re:Qwen4-Max by dtech on Monday November 16, 2026 @02:16AM (#50514937)
For what it's worth, in our own metrics and benchmarks the intelligence is a pretty good match, but the cost per task is wildly out of whack. We see the open weight models like GLM-5.3-flash or Deepseek 4.1 Flash 1017 use a fraction of their tokens/cost of the benchmarks, which changes the cost-benefit equation a lot.
Re:Qwen4-Max by anon373839 on Monday November 16, 2026 @02:19AM (#50514962)
> Probably because it’s hard to benchmaxxed against.

It's mainly just a composite of existing benchmarks. It would be very easy to benchmax if, say, AA shared with a preferred lab which benchmarks it plans to upweight/downweight in the next version. Just speaking hypothetically, of course.

Re:Qwen4-Max by puppymaster on Monday November 16, 2026 @02:33AM (#50515034)
Hence the constant switching around and running the benchmark themselves. They have been pretty transparent about which benchmarks get added or removed with every release.
Re:Qwen4-Max by rvba on Monday November 16, 2026 @02:20AM (#50514968)
or maybe it's the other way round, open weight models are under siege while closed weight models are defended
Re:Qwen4-Max by DeathArrow on Monday November 16, 2026 @03:24AM (#50515298)
If Artificial Analysis is not to be trusted, then which benchmarks are?
Re:Qwen4-Max by DCKing on Monday November 16, 2026 @04:07AM (#50515552)
I don't think there are any that can be trusted. There's too much riding on these benchmark results so the labs find ways to game them either by benchmaxing or by influencing the format of the benchmark.
Re:Qwen4-Max by ctolsen on Monday November 16, 2026 @04:25AM (#50515655)
Check others as well. https://www.vals.ai/ has it just under Astra.
Re:Qwen4-Max by greenchair on Monday November 16, 2026 @12:06PM (#50519899)
your own
Re:Qwen4-Max by spwa4 on Monday November 16, 2026 @05:04AM (#50515892)
I think you mean that benchmaxxing has turned into a business model, and artificial-analysis is not even the worst offender.

Right now the big labs are buying benchmark tasks from "benchmark labs" (or whatever they call themselves) for absurd amounts of money, and making sure they perform well on them.

The incentives this generates are terrible.

The big AI labs will only buy benchmark tasks that the frontier models perform poorly on (because they have trouble finding any public ones). So the tasks are being mass-written now directly based on the models they're supposed to train. Obviously the math says you shouldn't do this. If there is a causal connection between the benchmark and the model, you lose the normal guarantee that improving benchmark scores will improve model.

In fact improving such benchmarks may make the model worse. To understand that let's just give a ridiculous example. I create tasks like "1 3 5 11 13 17, give next number". The next number is 301. These are the house numbers on some random street in Brussels, Belgium (I'm guessing the big jump is because they "merged" 2 streets, and just added 300 to the numbers of the second part).

If the model trains on this, performance will actually decrease because obviously this is not how you should reason.

BUT making tasks under this "impossible to reason towards" part is of course how you make tasks that might even be reasonable, and of course frontier models are going to suck on tasks like this. So they sell!

Re:Qwen4-Max by Grimburger on Monday November 16, 2026 @08:13AM (#50517272)
> Favor Anthropic

Rarely seems to make them look good whenever I drop in to the site. Far more expensive for little gain. Even their own effort levels look bad, xhigh is 50% more in cost than high for zero improvement across a range of tasks.