Top model scores may be skewed by Git history leaks in SWE-bench

Re:Top model scores may be skewed by Git history leaks in SW... by TheDong on Thursday September 11, 2025 @06:17PM (#45216677)

I'm sure some of the people working at Theranos thought there legitimately was a revolutionary blood-test machine.

The presence of a person who wants SWE-bench to have honest results and takes it seriously does not mean the results are free of perverse incentives, nor that everyone is behaving just as honestly.

[ Parent | on HN ]

Re:Top model scores may be skewed by Git history leaks in SW... by Zacharias030 on Friday September 12, 2025 @05:25PM (#45226942)

When Swe-Bench was new in 2023, it was — with all due respect — a bit of a niche benchmark in LLM research. LLMs were so incredibly useless at solving these tasks that I think you could find a bit more empathy for the original academic authors. I don’t think the Theranos example applies. Even the flawed benchmark was good enough to get us from ~GPT4 to Claude 4‘s coding ability.

[ Parent | on HN ]