News for hackers. Stuff that matters.

A comment on Why SWE-bench Verified no longer measures frontier coding capabilities. Parent comment

Re:Why SWE-bench Verified no longer measures frontier coding... by cjsaltlake on Sunday April 26, 2026 @02:42PM (#47912708)
code clash I think would be quite hard to game or contaminate unintentionally; considering that models need to compete against one another
[ Parent | on HN ]
Re:Why SWE-bench Verified no longer measures frontier coding... by gertlabs on Sunday April 26, 2026 @03:05PM (#47912929)
https://gertlabs.com already does this at scale.

An industry-standard benchmark shouldn't be hosted or designed by a lab producing the models, regardless.

[ Parent | on HN ]
Re:Why SWE-bench Verified no longer measures frontier coding... by Bombthecat on Sunday April 26, 2026 @02:53PM (#47912803)
I mean the data / benchmarks
[ Parent | on HN ]