Re:Is Opus 5.5 nerfed? Thinking patterns changed on November 3
by peri-cl on Monday November 09, 2026 @05:02AM
(#50423479)
IMHO the only defensible way to benchmark this is to use secret-shoppers via subscriptions. As is standard practice in other product-review fields, where the stakes are a lot lower.
I *do* wish we wouldn't let software companies call this "A/B testing". That's not what that term means to a typical reader. In any other market than software, you'd call it something like "misleading consumers in respect to product composition" and "an unfair commercial practice". That's verbiage from the EU "dual quality" food scandal, where transnational food brands (Coca-Cola was one) adulterated their recipes to use cheaper ingredients in select markets, though the external packaging was kept the same.
I *do* wish we wouldn't let software companies call this "A/B testing". That's not what that term means to a typical reader. In any other market than software, you'd call it something like "misleading consumers in respect to product composition" and "an unfair commercial practice". That's verbiage from the EU "dual quality" food scandal, where transnational food brands (Coca-Cola was one) adulterated their recipes to use cheaper ingredients in select markets, though the external packaging was kept the same.
https://ec.europa.eu/commission/presscorner/detail/en/qanda_...
(I suppose Coca-Cola's CEO would prefer we talk about what they did as "A/B testing").