That must be on anthropic's benchmaxxing CI test

Nerfmaxxing.

It probably need a whistleblower to uncover what they are doing.

Nothing stopping Anthropic (or other labs) from pulling a VW Dieselgate and ensuring that benchmarks are routed to a non-nerfed model.

Its incredibly easy to roll out a compute instance to X% of users, or route traffic between compute instances

so not everyone will have the same experience. If I was saving on costs that would be the easiest way to do it without deploying, without any nefarious intent. The persons in charge of that likely don't even know about how individual users are impacted

The degradation only happens to some accounts but not all.

https://www.reddit.com/r/ClaudeCode/s/64DXRERLyU

It would be nice to know how they select. I max out my $200 plan every week and I was selected. On Saturday, the Opus on my account that happened to be supervising a running live process, became unusable.

It would be trivial for them to opt-out accounts that are doing benchmarking from a planned regression.

I really really really do not trust companies on the verge of IPOing from goosing their numbers. Even if the lawyers are up their ass, the social pressures are more than enough to compel the behavior. For every good person willing to do the right thing even at the cost of a delayed career progression, there are 20 people willing to do anything not even for a dollar but for an "atta boy!" from the boss.

That's a major reason to use local models. Or open weights models on a trusted cloud provider. Even if they are way less capable, they're more predictable

At what point does this become false advertising, or even fraud? Imagine paying AWS for a high-performance server, but 5% of customers who order one quietly get a slower, cheaper server and still pay the full price. That's a serious claim and makes me skeptical Anthropic is doing something like that without more evidence.

Actually, as a person who rented a dedicated server two (not from aws) this exact thing happens quite often.

For example a provider quietly underclocks your cpu or puts it is an energy saving plan hoping you won't notice.

"Nerf tracking" is adversarial and exploratory. It's not something Anthropic wants anyone to do, and it's not something that the "nerf tracking" designers are necessarily doing in a foolproof way. Don't assume too much from them either way.

But do from anecdotal evidence without benchmarks?

[delayed]

No, you did not comment on it. That is why I asked. Thanks for the clarification!

These nerf trackers do not work. I wish people would stop posting them.

They don't detect a quantized model until the quantization gets down to 4 bits or less.

Humans absolutely do detect quantization well before 4 bits...

Try it yourself if you want. Try a small local model, both quantized and not, and I would be very surprised if it isn't immediately obvious which is which.

But quantized local models also dip in evals.

It doesn't matter if it's local or not, I just suggested local so people could test themselves.

But no, evals and benchmarks do not detect quantization, until around 4 bit quantization.

https://developers.redhat.com/articles/2024/10/17/we-ran-ove...

https://pub.towardsai.net/your-model-passed-safety-evals-the...

The problem is that it seems to be an A/B testing thing:

https://news.ycombinator.com/item?id=50422951

So only if you were "selected" for their experiments, your account was nerfed.

But the claim here is "the models are nerfed before a release", not "they A/B test quants / efforts". If it were A/B testing, it doesn't really make sense to correlate it to release windows at all.