News for hackers. Stuff that matters.

A comment on Top model scores may be skewed by Git history leaks in SWE-bench. Parent comment

Re:Top model scores may be skewed by Git history leaks in SW... by Zacharias030 on Thursday September 11, 2025 @06:04PM (#45216582)
You realize that this problem in SWE-Bench was discovered and publicized by people within those FAANG corporations?
[ Parent | on HN ]
Re:Top model scores may be skewed by Git history leaks in SW... by TheDong on Thursday September 11, 2025 @06:17PM (#45216677)
I'm sure some of the people working at Theranos thought there legitimately was a revolutionary blood-test machine.

The presence of a person who wants SWE-bench to have honest results and takes it seriously does not mean the results are free of perverse incentives, nor that everyone is behaving just as honestly.

[ Parent | on HN ]
Re:Top model scores may be skewed by Git history leaks in SW... by Zacharias030 on Friday September 12, 2025 @05:25PM (#45226942)
When Swe-Bench was new in 2023, it was — with all due respect — a bit of a niche benchmark in LLM research. LLMs were so incredibly useless at solving these tasks that I think you could find a bit more empathy for the original academic authors. I don’t think the Theranos example applies. Even the flawed benchmark was good enough to get us from ~GPT4 to Claude 4‘s coding ability.
[ Parent | on HN ]