Why SWE-bench Verified no longer measures frontier coding capabilities

Re:Why SWE-bench Verified no longer measures frontier coding... by WarmWash on Sunday April 26, 2026 @11:44AM (#47911213)

The open models only give the SOTA models a run for their money on gameable benchmarks. On the semi-private ARC-AGI 2 sets they do absolutely awfully (<10% while SOTA is at ~80%)

It might be too expensive, but I would be interested in the benchmarks for the current crop of SOTA models.

[ Parent | on HN ]

Re:Why SWE-bench Verified no longer measures frontier coding... by roenxi on Sunday April 26, 2026 @12:45PM (#47911672)

Have the open models been tried? When I look at the leaderboard [0] the only qwen model I see is 235B-A22B. I wouldn't expect an MoE model to do particularly well, from what I've seen (thinking mainly of a leaderboard trying to measure EQ [1]) MoE models are at a distinct disadvantage to regular models when it comes to complex tasks that aren't software benchmark targets.

[0] https://arcprize.org/leaderboard

[1] https://eqbench.com/index.html

[ Parent | on HN ]

Re:Why SWE-bench Verified no longer measures frontier coding... by WarmWash on Sunday April 26, 2026 @02:24PM (#47912562)

There is GLM 5 and kimi 2.5 (which gets 11.8%, but I digress)

[ Parent | on HN ]