They also have this comment in getting started: "Pin the CLI. This is not optional: a Claude Code update changes the harness, and a changed harness looks exactly like a changed model. Turn off auto-updates and write down the version you're pinning"
Wild because as an end user I don't care what Anthropic changed to affect performance, I only care that Anthropic caused degraded performance.
So your harness behavior can change without updating it.
Fun.
If there was a silent degradation then the recent chatter about suspected nerfing is what you would expect to see. Of course that's not evidence, but I dont think we can say it's far more likely that its all subjective.
I mean the basic intuition doesn't seem unfouded. Maybe people can confirm/deny this contention for compute. But Anthropic is compute constrained. Inference takes compute. Model development takes compute. Isn't it plausible that there will be periods of contention? Or do we have information that suggests inference compute is affected by model development?
Anthropic has confirmed that they make changes to the models, for example with the recent update to limit reverse engineering.
[1] https://www.rensoart.com/why-your-eye-is-ahead-of-your-hand-...
The only thing I could maybe see is that any changes that are being made are happening on the subscription tiers on not the api tiers, but even that seems like quite a stretch.
Like anyone who is building agents with a good eval suite would know, models have been pretty consistent and even small harness changes can impact performance considerably.
If you calculate Elo only before 11/3 and compare to only after 11/3, the Elo is 52 points lower after. Unfortunately we are still in beta so this isn’t conclusive (confidence interval of -160 to +40).
If we do this calculation on all our models, Opus 5.5 is first in apparent drop. (Again, not definitive, but supports the conclusion.
He has his "nerfbench" he calls it. Very useful and entertaining at least. :)
Firstly, there is a lot of A/B testing. You would likely need at least 10 or 20 accounts to catch these random tests.
Secondly, the labs obviously run GPQA and MMLU like livenerf does and watch for their own regressions before doing A/B tests. These benchmarks are statistically the thing already being filtered out. You need tasks out of distribution. Ideally ones in distribution with the stuff they particularly frequently change (like safety or copywrite adjacent tasks).
These harnesses, and hidden system prompts, do get screwed with quite a lot in my experience. The models are actually remarkably consistent via API but there are tons of stealth system prompt and harness changes that lobotomize these models all the time.
Like despite being stochastic, try asking Luna to generate you a video game pitch via the API 100 times. It's almost always the same thing about lighthouses for some reason. That's actually why I tried to make a CLI entropy generator project.
The only one I'm familiar with testing Opus 5.5 specifically over time is https://github.com/ninjahawk/livenerf and it's not showing anything.