Re:Claude Haiku 5.5
by agentdev001 on Thursday October 08, 2026 @11:43AM
(#50007220)
I've considered this as well, as many others have. I tend to think that A: this is moreso a feature rather than a bug, and B: this is almost impossible to measure- it's a boogeyman imo.
In other words, combating this is impossible if it's happening. Measuring whether its happening is also not plausible, for small fish running bespoke benchmarks suites. I think we (consumers) have hit a pretty clear stride of; new model releases > some subset of evals/benchmarks are saturated > new, harder, more niche evals and benchmarks take their place. That doesnt seem unhealthy to me.
In other words, combating this is impossible if it's happening. Measuring whether its happening is also not plausible, for small fish running bespoke benchmarks suites. I think we (consumers) have hit a pretty clear stride of; new model releases > some subset of evals/benchmarks are saturated > new, harder, more niche evals and benchmarks take their place. That doesnt seem unhealthy to me.