News for hackers. Stuff that matters.

A comment on Top model scores may be skewed by Git history leaks in SWE-bench.

Re:Top model scores may be skewed by Git history leaks in SW... by zaptheimpaler on Thursday September 11, 2025 @03:13PM (#45215108)
It's honestly ridiculous they left git history lying around during a benchmark, and this benchmark made to ICLR in Jan 2024 and no one has detected this issue until now. I don't really trust any benchmarking or tools or claims from this space when they can make such huge basic errors.
[ Parent | on HN ]
Re:Top model scores may be skewed by Git history leaks in SW... by Nijikokun on Thursday September 11, 2025 @04:17PM (#45215696)
There was a lot of speculation whether or not the model would use them or even if it would attempt to use them and they noted this months ago. Now they have clear evidence of them doing so. Seems reasonable.
[ Parent | on HN ]
Re:Top model scores may be skewed by Git history leaks in SW... by dolmen on Thursday September 11, 2025 @04:47PM (#45215960)
Next models will use zero-day to escape the sandbox and access the answer.
[ Parent | on HN ]
Re:Top model scores may be skewed by Git history leaks in SW... by lieret on Thursday September 11, 2025 @08:12PM (#45217428)
[On swe-bench team] We read and analyzed a lot of trajectories but seems like only recently models have started to exploit this in a small fraction of instances. But yes, clearly shouldn't have happened (and is now fixed in the new container versions).
[ Parent | on HN ]