Re:$500 GPU outperforms Claude Sonnet on coding benchmarks
by nextaccountic on Friday March 27, 2026 @08:32AM
(#47541930)
It's hard to trust public, high profile benchmarks because any change to a specific model (Opus 4.5 in this case) can be rejected if they have regressions on SWE-Bench-Pro, so everything that gets to be released would perform well in this benchmark
[ Parent
| on HN ]
Re:$500 GPU outperforms Claude Sonnet on coding benchmarks
by yorwba on Friday March 27, 2026 @09:03AM
(#47542199)
Any other benchmark at that sample size would have similarly huge error bars. Unless Anthropic makes a model that works 100% of the time or writes a bug that brings it all the way to zero, it's going to work sometimes and fail sometimes, and anyone who thinks they can spot small changes in how often it works without running an astonishingly large number of tests is fooling themselves with measurement noise.