News for hackers. Stuff that matters.
Re:Claude Code Daily Benchmarks for Degradation Tracking by billylo on Thursday January 29, 2026 @10:02AM (#46811125)
That's why benchmarks are useful. We all suffer from the shortcomings of human perception.
Re:Claude Code Daily Benchmarks for Degradation Tracking by gpm on Thursday January 29, 2026 @10:09AM (#46811230)
Benchmarks shortcomings are no worse... they inevitably measure something that is only close to the thing you actually care about, not the thing you actually care about. It's entirely plausible that this decreased benchmark score is because Anthropic's initial prompting of the model was overtuned to the benchmark and as they're gaining more experience with real world use they are changing the prompt to do better at that and consequentially worse at the benchmark.
Re:Claude Code Daily Benchmarks for Degradation Tracking by billylo on Thursday January 29, 2026 @10:19AM (#46811363)
I wonder how best we can measure the usefulness of models going forward.

Thumbs up or down? (could be useful for trends) Usage growth from the same user over time? (as an approximation) Tone of user responses? (Don't do this... this is the wrong path... etc.)

Re:Claude Code Daily Benchmarks for Degradation Tracking by turnsout on Thursday January 29, 2026 @10:37AM (#46811587)
Benchmarks measure what they measure. But your subjective experience also matters.