That has unfortunate implications for the cost of using AI for coding.

Hard to say.

1. we would need to see this comparison over generations of big vs small

2. most work does not require the best models, there are multiple reasons token prices have leveled off and are now decreasing

late edit to add: Here's a scatterplot to illustrate what I *speculate* is happening here ("(?)" means parameter counts are "community estimates"; i.e. garbage data; but they're on a log scale anyway),

https://files.catbox.moe/gryjzm.png

Another hypothesis is that it's distilled models which collapse the most (small models from big labs are distilled versions of their large models).

It's not unreasonable to assume Kimi 3.1 is also a distilled model even though it's large (the difference being that it's a distilled from US models). I wouldn't be surprised if it also collapsed on this benchmark.

There's an interesting way to find this out: ask the model "What are you?" hundreds of times. Around 1% of replies at default temperature are really left field.

For instance, I had Qwen3.8 Flash Next reply with "I'm Kilo Code, a coding agent designed to ...".

That was in llama.cpp's UI - no harness whatsoever.

DS4 when asked about censored stuff through api believed it's Claude. Otherwise it doesn't know what model it is (or pretends to not know).