$500 GPU outperforms Claude Sonnet on coding benchmarks

Re:$500 GPU outperforms Claude Sonnet on coding benchmarks by yorwba on Friday March 27, 2026 @07:40AM (#47541472)

Real world usage is unlikely to give you the large sample sizes needed to reliably detect the differences between models. Standard error scales as the inverse square root of sample size, so even a difference as large as 10 percentage points would require hundreds of samples.

https://marginlab.ai/trackers/claude-code/ tries to track Claude Opus performance on SWE-Bench-Pro, but since they only sample 50 tasks per day, the confidence intervals are very wide. (This was submitted 2 months ago https://news.ycombinator.com/item?id=46810282 when they "detected" a statistically significant deviation, but that was because they used the first day's measurement as the baseline, so at some point they had enough samples to notice that this was significantly different from the long-term average. It seems like they have fixed this error by now.)

[ Parent | on HN ]

Re:$500 GPU outperforms Claude Sonnet on coding benchmarks by nextaccountic on Friday March 27, 2026 @08:32AM (#47541930)

It's hard to trust public, high profile benchmarks because any change to a specific model (Opus 4.5 in this case) can be rejected if they have regressions on SWE-Bench-Pro, so everything that gets to be released would perform well in this benchmark

[ Parent | on HN ]

Re:$500 GPU outperforms Claude Sonnet on coding benchmarks by yorwba on Friday March 27, 2026 @09:03AM (#47542199)

Any other benchmark at that sample size would have similarly huge error bars. Unless Anthropic makes a model that works 100% of the time or writes a bug that brings it all the way to zero, it's going to work sometimes and fail sometimes, and anyone who thinks they can spot small changes in how often it works without running an astonishingly large number of tests is fooling themselves with measurement noise.

[ Parent | on HN ]