News for hackers. Stuff that matters.
Re:Why SWE-bench Verified no longer measures frontier coding... by Escapado on Sunday April 26, 2026 @12:56PM (#47911768)
I agree with the sentiment but I wonder if a sufficiently large amount of sufficiently sophisticated benchmarks existed then I would be surprised if a model would only memorize those benchmarks while showing terrible real world performance. We are not there yet but maybe one day we will be.

There are no comments on this comment yet.