Sure, people complain about Anthropic's AI models getting worse over time. As well as OpenAI's models getting worse over time. But guess what? If you serve them open weights models, they also complain about models getting worse over time. Same exact checkpoint, same exact settings, same exact hardware.
Relative LMArena metrics, however, are fairly consistent across time.
The takeaway is that users are not reliable LLM evaluators.
My hypothesis is that users have a "learning curve", and get better at spotting LLM mistakes over time - both overall and for a specific model checkpoint. Resulting in increasingly critical evaluations over time.
Quantization could be done, not to deliberately make the model worse, but to increase reliability! Like Apple throttling devices - they were just trying to save your battery! After all there are regular outages, and some pretty major ones a handful of weeks back taking eg Opus offline for an entire afternoon.
> But guess what? If you serve them open weights models, they also complain about models getting worse over time.
Isn't this also anecdotal, or is there data informing this statement?
I think you could be partially right, but I also don't think dismissing criticism as just being a change in perspective is correct either. At least some complaints are from power users who can usually tell when something is getting objectively worse (as was the case for some of us Claude Code users recently). I'm not saying we can't fool ourselves too, but I don't think that's the most likely assumption to make.
Living evals can solve for the quantitative issues with infra and model updates, but not sure how to deal with perceptual adaptation.
People who like the tool at first use it until they stop liking it -> "it got worse"
People who dislike the tool at first do not use it -> "it was bad"
> I don't know how they could have phrased it that closed the door on this concern
Agreed. A full legal document would probably be the only way to convince everyone.
> we never intentionally degrade model quality as a result of demand or other factors
Fully giving them the benefit of the doubt, I still think that still allows for a scenario like "we may [switch to quantized models|tune parameters], but our internal testing showed that these interventions didn't materially affect end user experience".
I hate to parse their words in this way, because I don't know how they could have phrased it that closed the door on this concern, but all the anecdata (personal and otherwise) suggests something is happening.