I used to be skeptical of nerfing, but as of a week or so ago I've noticed a severe degradation of quality for GPT Astra. It struggled with very simple tasks, and it often failed to follow my instructions.

GPT Astra felt nerfed just days after it launched.

why were you skeptical of other users?

People are prone to seeing patterns in noise especially after suggestion. It's not hard to imagine someone concluding the model is nerfd after few bad responses in a row

Why this won’t apply to you as well?

i'm not parent, but to answer your snark : there are efforts a person can take to try to reduce things like suggestability in ones' own personality. Same as anything, the brain can be trained, thus it should go without saying that some of us react differently in the face of suggestion -- both for the better and the worse.

It could apply to me but I run evals every time someone at work complains about this and it’s never shown any more or less variance than regular usage. I don’t have statistical relevance with my evals but I do have a decent history with them.

People seem to underestimate how high normal variance is with these tools.

A growing percentage of the users on the internet are not human.

I hear a lot of accusations of nerfing as in, saying that the model itself has been quantized.

I doubt this is the case. I suspect all variation in quality and output can be explained in terms of prompt differences, and changing how much thinking happens at each level, as we see in the post above.

it's easy to suspect the hosting companies trying to do what they can to reduce cost-to-host up until the very threshold of perceptibility, so it's an easy guess to say they're probably doing something that shifts performance up or down between version releases..

but realistically I think you're right, it's prompt quality and the inherent non-determinism elements of LLMs showing up; when both of those things meet you get brain dead outputs.

Everyone likes to shout "slot machines! non-determinism!" until it actually explains something, heh.

Suppose a company has a limited amount of compute. What do they then do if peak demand exceeds what they can provide? They don't turn off people's access...

And what about A/B testing? Anthropic doesn't even try to hide that they do this. Limited compute and A/B testing alone, would guarantee a nerfed experience some of the time.

I consider it gas lighting to deny model nerfing.

I'm not denying model nerfing. I'm asserting an opinion about the nature of the nerfing.

If you want to go accuse someone of denying model nerfing, go find someone doing it and argue with them.

> They don't turn off people's access...

They do drop requests though? At peak times I sometimes have 1-2 requests fail with Opus; no biggie, the harness just retries it. But that, in aggregate, sheds load.

It's been the same for every version the last year.

Oh, this Opus X.Y is incredible. Hmm... Now it's dumb. Rinse, repeat.

Doesn't really matter if the cause is quantization, system prompt changes, changes to thinking etc.

It's hard to see how continuously rug pulling users at every release is a good idea. Only reason I can fathom is that it's needed to keep costs inline.

Either that or it's some weird tweaking to the prompts to fix whatever. But given the prevalent sentiment that it's worse that sounds like a line shot

Both Astra and Opus didn't really looked good for the last week or two.

Sol been really bad in some basic stuff that I would remove it from frontier models entirely

YES. To the point that I literally stopped using it, and have moved BACK to Claud.

This is getting ridiculous. The number of times I have switched from one model back to the other is crazy. I do not have any proof, so its quite possible I'm just delusional. AI psychosis? Either way, it FEELS like these models work really well for a while and then fall off. Switching to the latest model on the competitor always seems better, until it too seems to taper off. By then, a new model is usually available at the competitor, so I switch.

Its possible there's no degradation at all and just a PERCEPTION of it. But to me the end user, it really doesn't matter.

Yes I agree, Astra has been awful this week.

> Damn people were right about 5.5 being nerfed in preparation for a version release. This is not at all a good look.

Anthropic does this time and time again, unserious company

Do you also believe that they nerf the model being served by AWS Bedrock?

It gets worse. You might wonder "why now"? Well ... this happened 2 days ago:

https://x.com/follistatindev/status/2119129611510383014

With early handouts of Qwen4 starting to be reported 5 days ago ... so if they're good, they would have had some hours to evaluate Qwen4 against Claude 5.5 and, then or right after was when Anthropic decided to put out 5.2 ...

So yeah ... benchmaxxing and all that. BUT nevertheless, the published results seem to say that Qwen4-Max appears to match Opus 5.5 on some benchmarks, and Qwen4-27B (probably by going even harder than Qwen3.8-27B on thinking/inference time compute) can beat Sonnet 5.5 on some things.

I guess they're not taking any chances. Keep in mind that Anthropic has declared they have non-public models (it's in their IPO filing). These are referred to by the shorthand names Model 2, Model 3, and Model 4. Essentially 4 is better than 3 is better than 2.

Fable 5.2 is "2.5" so to speak. So Anthropic was holding back, but not anymore. And now they can dial the performance up once more with what they have ready, and probably once more after that.

Dario is gonna have to honeypot someone from Alibaba to get ahead.

Do you know of any rumors if they are still planning to release Fable 5.5 before the IPO or today's 5.2 is it?

“Why now” - they are about to IPO. It has been obvious for months that they would drop a new flagship model at exactly this point.

It would be funny though if you are right and that model is Fable 5.5 in a couple weeks.

it was so obvious that i even decided to get myself an anthropic sub today, hoping that i'd get a reset or something and i actually did

Plus the usual saying that new models are at its best for the first few days.

I dunno there are some other nerf trackers that aren't showing a regression: https://github.com/ninjahawk/livenerf

That must be on anthropic's benchmaxxing CI test

Nerfmaxxing.

It probably need a whistleblower to uncover what they are doing.

Nothing stopping Anthropic (or other labs) from pulling a VW Dieselgate and ensuring that benchmarks are routed to a non-nerfed model.

Its incredibly easy to roll out a compute instance to X% of users, or route traffic between compute instances

so not everyone will have the same experience. If I was saving on costs that would be the easiest way to do it without deploying, without any nefarious intent. The persons in charge of that likely don't even know about how individual users are impacted

The degradation only happens to some accounts but not all.

https://www.reddit.com/r/ClaudeCode/s/64DXRERLyU

It would be nice to know how they select. I max out my $200 plan every week and I was selected. On Saturday, the Opus on my account that happened to be supervising a running live process, became unusable.

It would be trivial for them to opt-out accounts that are doing benchmarking from a planned regression.

I really really really do not trust companies on the verge of IPOing from goosing their numbers. Even if the lawyers are up their ass, the social pressures are more than enough to compel the behavior. For every good person willing to do the right thing even at the cost of a delayed career progression, there are 20 people willing to do anything not even for a dollar but for an "atta boy!" from the boss.

That's a major reason to use local models. Or open weights models on a trusted cloud provider. Even if they are way less capable, they're more predictable

At what point does this become false advertising, or even fraud? Imagine paying AWS for a high-performance server, but 5% of customers who order one quietly get a slower, cheaper server and still pay the full price. That's a serious claim and makes me skeptical Anthropic is doing something like that without more evidence.

Actually, as a person who rented a dedicated server two (not from aws) this exact thing happens quite often.

For example a provider quietly underclocks your cpu or puts it is an energy saving plan hoping you won't notice.

"Nerf tracking" is adversarial and exploratory. It's not something Anthropic wants anyone to do, and it's not something that the "nerf tracking" designers are necessarily doing in a foolproof way. Don't assume too much from them either way.

But do from anecdotal evidence without benchmarks?

[delayed]

No, you did not comment on it. That is why I asked. Thanks for the clarification!

These nerf trackers do not work. I wish people would stop posting them.

They don't detect a quantized model until the quantization gets down to 4 bits or less.

Humans absolutely do detect quantization well before 4 bits...

Try it yourself if you want. Try a small local model, both quantized and not, and I would be very surprised if it isn't immediately obvious which is which.

But quantized local models also dip in evals.

It doesn't matter if it's local or not, I just suggested local so people could test themselves.

But no, evals and benchmarks do not detect quantization, until around 4 bit quantization.

https://developers.redhat.com/articles/2024/10/17/we-ran-ove...

https://pub.towardsai.net/your-model-passed-safety-evals-the...

The problem is that it seems to be an A/B testing thing:

https://news.ycombinator.com/item?id=50422951

So only if you were "selected" for their experiments, your account was nerfed.

But the claim here is "the models are nerfed before a release", not "they A/B test quants / efforts". If it were A/B testing, it doesn't really make sense to correlate it to release windows at all.

It's not true. People need to stop believing these conspiracy theories.

No it’s not nerfed. Another poster said it the best, your dopamine receiptors are nerfed after the new model hype wears off

I'm guilty of suspecting this.

If this turns out to be accurate, I think that would be a pretty unethical thing for Anthropic. Even if it is just a "we had to allocate compute to this new model release".

It should be fraud to swap out the model. We need a term for this, something like a bait-and-switch.

I believe fraud is a sufficient term. You claim to be selling X but you actually sold them Y. Bait and switch is a selling tactic to avoid fraud where they make you believe product is X and then when you go to buy they tell you X isnt available but they have Y. While also unethical, it is not fraud because they did tell you what it is you are buying before you buy it.

if they just straight up dont sell you something you bought then its fraud.

I think there is a ripe opportunity for expanding the definition of fraud to include changing the "ingredients" of a named product on the market. It is not limited here, we see it with food, and even in the medical field

(I will die on the hill that medical fraud was committed when mRNA therapy was used as a treatment for covid and it was 'sold' to the public as a "vaccine" despite not being the same thing as any vaccine before it. The definition of vaccine was expanded to include it without sifficient public disclosure to warrant informed consent imo. This is not a statement about conspiracy or the efficacy of vaccines or a moral judgement on whether it was for the greater good or not.. it was simply an act of fraud that robbed the public of proper informed consent by making people believe it was X when it was really Y.)

So yes, I think there is a large need for the public to at the very least put a strong handle on the ability to change what a product name means. It is a trivial problem to solve considering it is only a problem due fraud or fraud adjacent activities.

heres the solution: You have your product name, and then any changes to it REQUIRE a new version name. Like how you have Coke and Diet Coke, only formalized such that Coke cannot legally replace the ingredients of regular Coke with the ingredients of Diet coke and still call it Coke. With Software they could at least use version numbers and release modifiers. most of the time its built in already, but it needs to include the full stack. if they change anything that has an interaction with the user it should change the label seen by the user for the product.

thank you for coming to my ted talk

You don't buy a hard quantity of an LLM you buy access in general.

A Opus 5.5 is not a Opus 5.5. Its Opus 5.5 of some checkpoint.

mRNA btw. was well known how it worked, it wasn't hidden and some people waited for the non mRNA version because of it. I never seen or read the argument you make and I do not get it.

A vaccine is something which protects you from an infection. What definition do you think changed in humans? That we make the protein? How is that so different from viral vector vaccines?

The consent is a certain level of trust from the society to a group of researchers limited by being humans and having to act in a certain time frame.

Nontheless, I do agree I would like to have a fingerprint and proper changelogs etc. for products I buy including digital and analog. But this has a downside as usual: The smaller you are, the more it costs you to (probably) register, maintain and change.

You know the good thing about Claude etc. right now? It doesn't matter. My invest is not 4 digits and there are that many changes, i'm quite happy to have been upgraded for 'free' from Opus 4.8 to Opus 5 (what do you call this upgrade, which was significant better?)

I've been working with Opus 5.5 every day, probably around 10h on average, and I feel confident in saying that they did not replace anything.

I'm a bit disappointed with how this nonsense is spread across HN today.

"I'm a bit disappointed with how this nonsense is spread across HN today."

Are you saying that the many, many people posting and experiencing this are wrong and you are right?

And not just in this instance, nerfing has been complained about in a cyclical fashion for many months.

But, they're all wrong?

You're so confident in your experience that you cannot think that other's experiences are also true?

I'm pretty convinced that those who have constantly been posting these complaints, probably for every single model release out there, are indeed wrong.

If you go back in time far enough, your logic can successfully be used to demonstrate the earth is flat.

"Could all these people seriously be wrong?" Yes, yes they could.

> And not just in this instance, nerfing has been complained about in a cyclical fashion for many months.

No, it gets complained about constantly even if there is no model coming out. A completely unrelated thread on Claude will feature comments like "can I just say that [model] has suddenly become useless this week?" Merely asking that question is bound to get people agreeing with you because we are evaluating all of this on vibes and placebos.

I too feel that way sometimes, I don't think it's ridiculous that Anthropic might silently downgrade you as demand warrants. But asking others if they "feel" something I feel is less than useless.

I literally hit the desk with an open hand yelling I KNEW IT

But, it’s okay they do it because they charge less when it happens.

Right?

idk, it hasn't felt nerfed at all to me, and it's not like i'm just doing trivial CRUD webapps either. opus 5.5 is very successfully porting my phone to postmarketos. claude just got the touchscreen up and working (and made a little program for me to test it, and it's legit).

Another reason we need to democratize local models.

I at least didn't see this. I had a baseline evaluation from before Nov 3 on Opus 5.5 medium, so I re-ran it this morning.

Eval context: 30 bug fixes and features evaluated by tests the agent didn't see. Ran on Opus 5.5 at medium effort in Claude Code.

20 of the tasks were last run on Nov 1, twice each. Nov 1 solved 36/40, and today solved 19/20. Thinking tokens per task were 0.95x Nov 1’s and output tokens 0.97x. The one significant change was speed: median time to first token decreased from 3.0s to 1.7s.

Caveats: one run per task today, medium effort only.

If I read your results correct it solved 5% more problems and used up between 3-5% fewer tokens, with a faster response time?

Correct. But the 5% more solves and 3-5% fewer tokens are within the noise of previous eval runs I've done on models. The faster response time is not.

Someone using an actual statistical test to quantify if there's actual measurable improvement?! An angel among men.

from my basic stats knowledge, it's not reliable to calculate the confidence interval of the sample proportion with fewer than about 10 samples on both sides. So if you're only seeing 1 or 2 failures in 20, and that approximately held as you scaled, you would have to run about 200 tests with each in order to get at least 10 failures in order to calculate the confidence intervals of the two samples and see if they overlap or not.

And this is why not many people do this. Even if you have the decals and the statistical knowledge, it's pretty expensive to do this regularly.

Uncertainty from count statistics are far larger than this. These are two perfectly consistent results there’s no statsig difference

The spending limit definitely goes down much faster as of last week or so.

Personally, I've found myself getting lazier recently, trying to get a single session to handle tasks that should be broken into multiple, etc. I've suspected claims of nerfing might be from changes in user habit. I'm glad more disciplined checks are being run.

It could be harness changes too, been quite unhappy with opus 5.5 the last few days and there have been a lot of harness updates.

No they weren't. Models are not "nerfed".

They do change things though which can have impact on behaviour.

System prompt would be the easiest one.

But there were flash or inbetween models, different checkpoints etc. THey might also try different hardware which might not be binary compatible.

People have been complaining about how the models are nerfed non-stop for two years. A stopped clock is right twice a day.

Benchmarks don't do a good job at capturing the difference in quality of frontier models over smaller models.

Opus 5.5 was so good that if it wasn't nerfed - at least just looking at the benchmarks - the difference in cost here does not seem to justify using this for anything besides design and review.

That's like maybe 1% of total token usage for SWE BUT a better design can have enormous benefits, so it's almost always worth using the best model for that.

I'm not surprised at all that they might be doing shady shit, my free request was used up in less than 7 minutes and said pay to continue, no thanks. I've paid for monthly before, never again I don't care how good the model is. I asked about the extreme limited usage in their discord and got banned, not a good look. You don't have a way to measure what you're actually paying for and they can change it at any time.

yeah, we need to make sure we don't get stuck with these two evil corps. Dario is really the worst. Plays the drama queen but he is very hard in all things dark(i.e. dark patterns).

It's a good pattern to detect for perhaps.

This happened last time too with Opus 5.5 (released September 22): https://news.ycombinator.com/item?id=49789224

This has happened every single time for the past year, why are people surprised? This is all so that the new models can look better on the benchmarks.

if this is their common practice, that's so shameful, nerfing before a new product release

That viral Tweet was analyzing the summarizer output, not Opus output.

The reasoning traces aren't available. The summarizer gives a short summary of what the reasoning traces do.

The changes were in the summarizer model. The Tweet doesn't even say that Opus was nerfed, everyone just assumed it because the summarizer started using "I'm..." more.

This mystery was solved on Twitter over the weekend. Watching the pitchforks come out on Hacker News on Monday over a situation that was already resolved and debunked is starting to feel like how LinkedIn catches on to internet drama about a week later.

I don't have a formal benchmark, but daily work (coding mostly, plus some complex architecture design and review) has had consistent outputs today for me. No noticeable nerf from my perspective. I definitely took notice in some past instances, but if they nerfed Opus 5.5, then I guess it was just so good that I was nowhere near its limits.

In a normal world you are not allowed to degrade the service during your agreed and paid contract. You would get sued. Apparently this is perfectly fine in this current world.

Can you share some evidence?

for me opus 5.5 was acting kind of dumb, like forgetting basic stuff he was nailing last week

Interesting - I had the same feeling but mostly with 4.6 when I "downgraded" for some easier/trivial tasks: The model forget things I posted 20 min ago in the chat (and the same file was in the context as well)

Cost per task

Fable 5.1 Max $11.46

Fable 5.2 Max $44.70