News for hackers. Stuff that matters.

A comment on Top model scores may be skewed by Git history leaks in SWE-bench.

Re:Top model scores may be skewed by Git history leaks in SW... by slacktivism123 on Thursday September 11, 2025 @03:17PM (#45215142)
Fascinating case showing how LLM promoters will happily take "verified" benchmarks at their word.

It's easy to publish "$NEWMODEL received an X% bump in SWE-Bench Verified!!!!".

Proper research means interrogating the traces, like these researchers did (the Gist shows Claude 4 Sonnet): https://gist.github.com/jacobkahn/bd77c69d34040a9e9b10d56baa...

Commentary: https://x.com/bwasti/status/1963288443452051582, https://x.com/tmkadamcz/status/1963996138044096969

[ Parent | on HN ]
Re:Top model scores may be skewed by Git history leaks in SW... by k__ on Thursday September 11, 2025 @04:41PM (#45215926)
Yes, often you see huge gains in some benchmark, then the model is ran through Aider's polyglot benchmark and doesn't even hit 60%.
[ Parent | on HN ]
Re:Top model scores may be skewed by Git history leaks in SW... by Workaccount2 on Thursday September 11, 2025 @05:07PM (#45216116)
The best benchmark is the community vibe in the weeks following a release.

Claude benchmarks poorly but vibes well. Gemini benchmarks well and vibes well. Grok benchmarks well but vibes poorly.

(yes I know you are gushing with anecdotes, the vibes are simply the approximate color of gray born from the countless black and white remarks.)

[ Parent | on HN ]
Re:Top model scores may be skewed by Git history leaks in SW... by wubrr on Thursday September 11, 2025 @06:00PM (#45216556)
the vibes are just a collection anecdotes
[ Parent | on HN ]
Re:Top model scores may be skewed by Git history leaks in SW... by ryoshu on Thursday September 11, 2025 @08:00PM (#45217354)
"qual"
[ Parent | on HN ]
Re:Top model scores may be skewed by Git history leaks in SW... by diggan on Friday September 12, 2025 @09:09AM (#45221729)
> The best benchmark is the community vibe in the weeks following a release.

True, just be careful what community you use as a vibe-check. Most of the mainstream/big ones around AI and LLMs basically have influence campaigns run against them, are made of giant hive-minds that all think alike and you need to carefully asses if anything you're reading is true or not, and votes tend to make it even worse.

[ Parent | on HN ]
Re:Top model scores may be skewed by Git history leaks in SW... by theblazehen on Friday September 12, 2025 @11:37AM (#45223309)
I generally check LM Arena as well as which models have had the most weekly tokens on openrouter
[ Parent | on HN ]