Top model scores may be skewed by Git history leaks in SWE-bench

Re:Top model scores may be skewed by Git history leaks in SW... by bflesch on Thursday September 11, 2025 @05:30PM (#45216312)

Unfortunately the bank account trajectories are not public, because unscupulous corporations such FAANG who let thousands of engineers wade through my chat messages on their platforms might not shy away from bribing academics to improve benchmarks of their billion-dollar AI initiatives.

It's also a bribe if my sibling gets a job with $500k annual salary. Tech is not immune to it.

[ Parent | on HN ]

Re:Top model scores may be skewed by Git history leaks in SW... by Zacharias030 on Thursday September 11, 2025 @06:04PM (#45216582)

You realize that this problem in SWE-Bench was discovered and publicized by people within those FAANG corporations?

[ Parent | on HN ]

Re:Top model scores may be skewed by Git history leaks in SW... by TheDong on Thursday September 11, 2025 @06:17PM (#45216677)

I'm sure some of the people working at Theranos thought there legitimately was a revolutionary blood-test machine.

The presence of a person who wants SWE-bench to have honest results and takes it seriously does not mean the results are free of perverse incentives, nor that everyone is behaving just as honestly.

[ Parent | on HN ]

Re:Top model scores may be skewed by Git history leaks in SW... by Zacharias030 on Friday September 12, 2025 @05:25PM (#45226942)

When Swe-Bench was new in 2023, it was — with all due respect — a bit of a niche benchmark in LLM research. LLMs were so incredibly useless at solving these tasks that I think you could find a bit more empathy for the original academic authors. I don’t think the Theranos example applies. Even the flawed benchmark was good enough to get us from ~GPT4 to Claude 4‘s coding ability.

[ Parent | on HN ]