GPT Astra felt nerfed just days after it launched.

why were you skeptical of other users?

People are prone to seeing patterns in noise especially after suggestion. It's not hard to imagine someone concluding the model is nerfd after few bad responses in a row

Why this won’t apply to you as well?

i'm not parent, but to answer your snark : there are efforts a person can take to try to reduce things like suggestability in ones' own personality. Same as anything, the brain can be trained, thus it should go without saying that some of us react differently in the face of suggestion -- both for the better and the worse.

It could apply to me but I run evals every time someone at work complains about this and it’s never shown any more or less variance than regular usage. I don’t have statistical relevance with my evals but I do have a decent history with them.

People seem to underestimate how high normal variance is with these tools.

A growing percentage of the users on the internet are not human.

I hear a lot of accusations of nerfing as in, saying that the model itself has been quantized.

I doubt this is the case. I suspect all variation in quality and output can be explained in terms of prompt differences, and changing how much thinking happens at each level, as we see in the post above.

it's easy to suspect the hosting companies trying to do what they can to reduce cost-to-host up until the very threshold of perceptibility, so it's an easy guess to say they're probably doing something that shifts performance up or down between version releases..

but realistically I think you're right, it's prompt quality and the inherent non-determinism elements of LLMs showing up; when both of those things meet you get brain dead outputs.

Everyone likes to shout "slot machines! non-determinism!" until it actually explains something, heh.

Suppose a company has a limited amount of compute. What do they then do if peak demand exceeds what they can provide? They don't turn off people's access...

And what about A/B testing? Anthropic doesn't even try to hide that they do this. Limited compute and A/B testing alone, would guarantee a nerfed experience some of the time.

I consider it gas lighting to deny model nerfing.

I'm not denying model nerfing. I'm asserting an opinion about the nature of the nerfing.

If you want to go accuse someone of denying model nerfing, go find someone doing it and argue with them.

> They don't turn off people's access...

They do drop requests though? At peak times I sometimes have 1-2 requests fail with Opus; no biggie, the harness just retries it. But that, in aggregate, sheds load.

You serve the requests more slowly. Same as if your bank has a sudden rush of customers - the tellers serve them in order, and the customers wait longer. Some of the customers 'time out' (get frustrated and leave).

It's been the same for every version the last year.

Oh, this Opus X.Y is incredible. Hmm... Now it's dumb. Rinse, repeat.

Doesn't really matter if the cause is quantization, system prompt changes, changes to thinking etc.

It's hard to see how continuously rug pulling users at every release is a good idea. Only reason I can fathom is that it's needed to keep costs inline.

Either that or it's some weird tweaking to the prompts to fix whatever. But given the prevalent sentiment that it's worse that sounds like a line shot

Both Astra and Opus didn't really looked good for the last week or two.

Sol been really bad in some basic stuff that I would remove it from frontier models entirely

YES. To the point that I literally stopped using it, and have moved BACK to Claud.

This is getting ridiculous. The number of times I have switched from one model back to the other is crazy. I do not have any proof, so its quite possible I'm just delusional. AI psychosis? Either way, it FEELS like these models work really well for a while and then fall off. Switching to the latest model on the competitor always seems better, until it too seems to taper off. By then, a new model is usually available at the competitor, so I switch.

Its possible there's no degradation at all and just a PERCEPTION of it. But to me the end user, it really doesn't matter.

Yes I agree, Astra has been awful this week.