Why SWE-bench Verified no longer measures frontier coding capabilities

Re:Why SWE-bench Verified no longer measures frontier coding... by cjsaltlake on Sunday April 26, 2026 @02:46PM (#47912748)

SWE-bench was created to replace olympiad coding benchmarks. I think past olympiad coding benchmarks were much worse representative of real-world coding than something like SWE-bench, which is derived from real units of labor.

Further, olympiad style benchmarks are arguably easier to contaminate / memorize unless you refresh it regularly; but that goes for SWE-bench too.

[ Parent | on HN ]

Re:Why SWE-bench Verified no longer measures frontier coding... by rustyhancock on Sunday April 26, 2026 @03:52PM (#47913515)

I was picturing one-shot performance only for the benchmark, on novel real world tasks. I.e. the score on the March Olympiad you got in April isn't relevant.

Simple enough that anyone could run it with a regular subscription.

Really unless we can get the providers to ditch the gameable benchmarks they won't.

But industries love nothing more than a benchmark they can manipulate.

[ Parent | on HN ]