This has happened to consistently across OpenAI and Anthropic models that I can't believe people are still denying it. Obviously they're silently quantising their models or making other efficiency changes. If they can save 40% RAM and only hurt intelligence by a few percent in their benchmarks, they're going to do that. Unfortunately their benchmarks are obviously not representative of the real world, and these downgrades are very noticeable for people who rely on these models for work.

Of all the reasons to look at local inference, this is the one which pushes me in that direction the most. I want to rely on a model for consistency in my workflows. If I wake up on Monday and suddenly it's refactoring things it built on Friday, that's a huge problem. That makes it unreliable.

[delayed]

Little of column A, little of column B.

These machines are inherently unreliable.

> I can't believe people are still denying it

Probably because those "nerfs" do not show on sites that track model performance over time. Sometimes Anthropic/OpenAI mess stuff up in their harness, but usually not in the models themselves.

It's easy to imagine they can detect when they're being profiled and route accordingly. Paranoia or not, it's possible.

People are denying it because it's simply not true and for sure has not happened consistently.

In my own work I have not once noticed a degradation where I could have said with any certainty that it's not within usual deviation or me having a bad day.

Conspicuously the model seems to be especially dumb and annoying when I did not get a lot of sleep the previous night.

When OpenAI and Anthropic engineers post here on HN that they do not quantize models and silently replace checkpoints in the API, I believe them.

I saw nothing that convinced me the people who are constantly making these complaints don't just mistake any suboptimal result for changes in the model.

It’s also true that people that perform their own evals see wide variance uncorrelated with “nerfing”. Human cognitive bias makes them only see that variance at specific times and then they start projecting.

Everyone really should keep a few tasks around that are indicative of the work they do and run them a few times with the same model and a few times across model families and when new models drop.

It’s extremely informative to see with your own eyes the variance that always exists. Even if you can’t afford statistical relevance it’s valuable.

I've "called" the last 3 out of 4 new model releases. Maybe my account it perma-tagged as one that gets downgraded to re-allocate resources before a release but about 5-7 days before a new model Claude suddenly gets dumber and dumber on executing canned skills that I've used for 6+ months with no issues. The cynic in me thinks they are dumbing down models right before a release so the new model seems much smarter.

Yes I have similar anecdotal experiences - it gets dumb before a new model. What I imagine in my head is Anthropic using all their GPUs to polish off the next model and they need to borrow GPU cycles so they cut down the thinking.