News for hackers. Stuff that matters.

A comment on Top model scores may be skewed by Git history leaks in SWE-bench. Parent comment

Re:Top model scores may be skewed by Git history leaks in SW... by jsheard on Thursday September 11, 2025 @03:07PM (#45215041)
> So the "Verified" part of "SWE Bench Verified" means.. not "Verified" at all.

Seems on-brand for an LLM-related thing to claim that it has verified something without actually checking.

[ Parent | on HN ]
Re:Top model scores may be skewed by Git history leaks in SW... by geekymartian on Thursday September 11, 2025 @03:29PM (#45215260)
that was my exact thought. how fitting
[ Parent | on HN ]
Re:Top model scores may be skewed by Git history leaks in SW... by hhh on Friday September 12, 2025 @08:55AM (#45221600)
Verified has a completely different meaning for this, it's that the questions have verified valid solutions.
[ Parent | on HN ]