Re:Is Opus 5.5 nerfed? Thinking patterns changed on November 3
by MallocVoidstar on Monday November 09, 2026 @04:32AM
(#50423238)
If Anthropic are doing A/B testing it almost certainly only affects subscription users, not API users. Every benchmark I know of uses the API.
Re:Is Opus 5.5 nerfed? Thinking patterns changed on November 3
by dstu on Monday November 09, 2026 @04:52AM
(#50423398)
If I were a cynic I’d also say that it wouldn’t be particularly difficult for them to detect when a benchmark/eval context is active and to route those requests to larger models or higher quants.
Re:Is Opus 5.5 nerfed? Thinking patterns changed on November 3
by peri-cl on Monday November 09, 2026 @05:02AM
(#50423479)
IMHO the only defensible way to benchmark this is to use secret-shoppers via subscriptions. As is standard practice in other product-review fields, where the stakes are a lot lower.
I *do* wish we wouldn't let software companies call this "A/B testing". That's not what that term means to a typical reader. In any other market than software, you'd call it something like "misleading consumers in respect to product composition" and "an unfair commercial practice". That's verbiage from the EU "dual quality" food scandal, where transnational food brands (Coca-Cola was one) adulterated their recipes to use cheaper ingredients in select markets, though the external packaging was kept the same.
https://ec.europa.eu/commission/presscorner/detail/en/qanda_...
(I suppose Coca-Cola's CEO would prefer we talk about what they did as "A/B testing").
I have no doubt they will ensure their users get an accurate picture of Opus 5.5 within hours.