AI Labs actually. They would love the perfect benchmar representing real usecase. I'd even dare and guess that they don't share much publically because it is a huge competitive advantage.

> they would love the perfect benchmar

That right there is the problem.

What is the problem?

ks

It is "a" problem, but honestly not a huge one. Chasing benchmarks is an integral part of AI research, and it is how we have gotten this far, which is pretty damn far. Discovering the ways in which benchmarks fail is in itself an interesting process, and the amount of interesting new benchmarks has grown with the capabilities of the models as a result. The frontier labs might be a bit selective about which benchmarks they publish as marketing, but overall they have so far been reasonably diligent about avoiding outright cheating. Most of the truly benchmaxed models are not on the frontier (eg. see muse spark performance dropping off a cliff in the article).

This is a scenario where the interest of the users and the labs actually align perfectly, since labs that have people actually paying for their product want benchmarks to align as close as possible to real life performance. Otherwise they can't actually easily tell whether the new model they are developing is any good or not, and so risk wasting literal billions of dollars on training runs for a product that people won't end up paying for. A failed major training run is an absolute disaster at the frontier, GPT-4.5 for instance was an almost existential failure for OpenAI at the time.

Case in point, the most popular coding model, Opus 5.5, also just happens to be the only one that avoided the benchmaxing impulse in this article. This is the result of years of Anthropic working very hard to avoid it.

Your last paragraph undermines the first two.

The benchmark makers and anyone who relies on benchmarks

depends, are they optimizing for accuracy or something else?

The people who have no power over the situation: users.

Users always have the choice to ignore benchmarks or run their own

Exactly. Even a brief one (a single task done on two models) showed me GLM 5.3 Flash was superior to DeepSeek 4.1 Flash.

DS used nearly twice the budget to accomplish the same thing.

Ideally, these tasks should be something you actually need day to day.

I'd rank it DeepSeek 4.1 Flash 1017 > GLM 5.3 Flash > DeepSeek 4.1 Flash (Original) - 1017 is a huge improvement.

Oh! I thought I had tested 1017, but I forgot about it. Thank you!

That’s missing the point. Frontier models get shittier in certain ways with each release, due to benchmaxxing, and there is nothing we can do about that.