News for hackers. Stuff that matters.

A comment on Why SWE-bench Verified no longer measures frontier coding capabilities. Parent comment

Re:Why SWE-bench Verified no longer measures frontier coding... by red75prime on Sunday April 26, 2026 @02:53PM (#47912796)
Desarcasmed version: "I think that problems with Zork make those models virtually useless in programming tasks." Correct?
[ Parent | on HN ]
Re:Why SWE-bench Verified no longer measures frontier coding... by cptskippy on Sunday April 26, 2026 @06:52PM (#47915624)
He said complicated code bases. LLMs are great at producing small snippets of code to address very targeted problems.
[ Parent | on HN ]
Re:Why SWE-bench Verified no longer measures frontier coding... by red75prime on Sunday April 26, 2026 @09:17PM (#47916692)
Great on small snippets of code, passable on larger pieces of code, great at finding vulnerabilities in large pieces of code, terrible in Zork. All-in-all, a jagged frontier that defies a simple sarcastic characterization.
[ Parent | on HN ]
Re:Why SWE-bench Verified no longer measures frontier coding... by girvo on Monday April 27, 2026 @08:28AM (#47920718)
Very kiki, not very bouba, as Aphyr rightfully stated.
[ Parent | on HN ]