Claude Code Daily Benchmarks for Degradation Tracking

Re:Claude Code Daily Benchmarks for Degradation Tracking by nikcub on Thursday January 29, 2026 @05:54PM (#46818011)

> I would run on 300 tasks and I'd run the test suite 5 or 10 times per day and average that score.

assume this is because of model costs. anthropic could either throw some credits their way (would be worthwhile to dispel the 80 reddit posts a day about degrading models and quantization) or OP could throw up a donation / tip link

[ Parent | on HN ]

Re:Claude Code Daily Benchmarks for Degradation Tracking by simsla on Thursday January 29, 2026 @06:56PM (#46818707)

Probably, but with a small sample size like that, they should probably be taking the uncertainty into account, because I wouldn't be surprised if a lot of this variation falls within expected noise.

E.g. some binomial interval proportions (aka confidence intervals).

[ Parent | on HN ]

Re:Claude Code Daily Benchmarks for Degradation Tracking by phist_mcgee on Thursday January 29, 2026 @05:57PM (#46818054)

Then you'd get people claiming that the benchmarks were 'paid for' by anthropic

[ Parent | on HN ]

Re:Claude Code Daily Benchmarks for Degradation Tracking by nikcub on Thursday January 29, 2026 @06:05PM (#46818137)

one thing you learn from being on the internet is that you're never going to satisfy everybody

[ Parent | on HN ]