I am more skeptical about the compute provider claim - do you have any evidence of that?
and there's sometimes just huge floods of complaints from people all of a sudden, which is pretty unlikely to be a coincidence
Where?
There are no bugs, just happy little accidents.
That just means they don’t reduce model quality for those reasons.
They didn’t mention other reason, for example, “Make more money”.
Christ this forum has become intellectually dishonest.
> There are recorded cases of real regressions, but they're better characterised as incidents, not nerfs
So I found this incident, as they called it, to still be relevant and why benchmarks such as the OP are useful.
It’s not just ant. There are so many small knobs that providers can claim isn’t nerfing but “load management” or “improving user experience”. One example from OpenAI is reducing juice to reduce time to first token.
Unintentional tbf.
fwiw, i think almost all regressions are down to a/b testing in the harness by anthropic, but it is objectively indistinguishable beyond "the coding agent ceases to be usable" and i'm back to traditional coding for a few hours until its back to normal again
It absolutely matters because something like Claude Code has no guarantee that there won't be changes between updates but a model pinned at the API version level that is getting enterprise traffic absolutely does have that guarantee and would be a much more widespread problem...
The /r/antigravity SubReddit is full of users who very much notice bugs with the tool/agent. We should be thankful that Claude Code is pretty stable by comparison.
Anthropic has admitted to nerfing in the past. There have also been inference bugs. On top of that, model performance changes as they move compute to schwaggier providers as well.
Your chart is wrong.