News for hackers. Stuff that matters.

A comment on SOTA on swebench-verified: relearning the bitter lesson.

Re:SOTA on swebench-verified: relearning the bitter lesson by attentive on Wednesday January 08, 2025 @05:18PM (#42639081)
This bench seems to be entirely python based. Are there similar benchmarks that test different languages for these tools?
[ Parent | on HN ]
Re:SOTA on swebench-verified: relearning the bitter lesson by ofirpress on Thursday January 09, 2025 @10:20AM (#42646506)
I'm one of the co-authors of SWE-bench. We just created a Javascript (+visual) SWE-bench: https://www.swebench.com/multimodal.html

We're going to release the eval suite for this soon so that people can start making submissions.

[ Parent | on HN ]