But admitting their model is weaker is a no go. So conspiracy theory here we go
It's mainly just a composite of existing benchmarks. It would be very easy to benchmax if, say, AA shared with a preferred lab which benchmarks it plans to upweight/downweight in the next version. Just speaking hypothetically, of course.
Right now the big labs are buying benchmark tasks from "benchmark labs" (or whatever they call themselves) for absurd amounts of money, and making sure they perform well on them.
The incentives this generates are terrible.
The big AI labs will only buy benchmark tasks that the frontier models perform poorly on (because they have trouble finding any public ones). So the tasks are being mass-written now directly based on the models they're supposed to train. Obviously the math says you shouldn't do this. If there is a causal connection between the benchmark and the model, you lose the normal guarantee that improving benchmark scores will improve model.
In fact improving such benchmarks may make the model worse. To understand that let's just give a ridiculous example. I create tasks like "1 3 5 11 13 17, give next number". The next number is 301. These are the house numbers on some random street in Brussels, Belgium (I'm guessing the big jump is because they "merged" 2 streets, and just added 300 to the numbers of the second part).
If the model trains on this, performance will actually decrease because obviously this is not how you should reason.
BUT making tasks under this "impossible to reason towards" part is of course how you make tasks that might even be reasonable, and of course frontier models are going to suck on tasks like this. So they sell!
Rarely seems to make them look good whenever I drop in to the site. Far more expensive for little gain. Even their own effort levels look bad, xhigh is 50% more in cost than high for zero improvement across a range of tasks.
They have altered their benchmark multiple times in the last 4 weeks (v5 in https://artificialanalysis.ai/articles/artificial-analysis-i...) and the last 5.0.1 change https://artificialanalysis.ai/methodology/intelligence-bench... pushed up Anthropic's models a bit, you can see that v5.0.0 set on the first link Opus 5.5 has a lower score than in v5.0.1, and v5.0.1 was released just before Fable dropped.
Some part of me believes that the benchmark mix was put together to favor Anthropic's models.