News for hackers. Stuff that matters.
Re:Claude Haiku 5.5 by doktor_pepsi on Thursday October 08, 2026 @04:45AM (#50003374)
Appreciate providing this benchmark. I am curious, do you think that as new models (from the same company) are released and you repeat the benchmark, they adapt/extend their training data to include your dataset, thus polluting the benchmark results? Would you be doing anything to combat this?
Re:Claude Haiku 5.5 by chriddyp on Thursday October 08, 2026 @10:15AM (#50006081)
It's a valid concern. We're not releasing the dataset, the answers, or the full set of questions to help prevent this. At the same time, I like to share where it gets things wrong in a bit more detail, which involves sharing a bit of the exam. I expect that we'll create a new benchmark with a new dataset in 6 months.
Re:Claude Haiku 5.5 by agentdev001 on Thursday October 08, 2026 @11:43AM (#50007220)
I've considered this as well, as many others have. I tend to think that A: this is moreso a feature rather than a bug, and B: this is almost impossible to measure- it's a boogeyman imo.

In other words, combating this is impossible if it's happening. Measuring whether its happening is also not plausible, for small fish running bespoke benchmarks suites. I think we (consumers) have hit a pretty clear stride of; new model releases > some subset of evals/benchmarks are saturated > new, harder, more niche evals and benchmarks take their place. That doesnt seem unhealthy to me.