Re:Why SWE-bench Verified no longer measures frontier coding...
by cjsaltlake on Sunday April 26, 2026 @02:37PM
(#47912672)
If you read the mythos report, in which they discuss and account for contamination substantially, it still suggests that performance on SWE-bench verified is meaningful. Benchmarks, including SWE-bench can absolutely be gamed, but if you're not explicitly benchmaxxing, improving on SWE-bench still measures model improvements, at least up to the level of Mythos.
[ Parent
| on HN ]
There are no comments on this comment yet.