News for hackers. Stuff that matters.

A comment on A Research Preview of Codex. Parent comment

Re:A Research Preview of Codex by ofirpress on Friday May 16, 2025 @08:46PM (#44011138)
Not sure what you mean by benchmaxxing but we think there's still a lot of useful signals you can infer from SWE-bench-style benchmarking.

We also have SWE-bench Multimodal which adds a twist I haven't seen elsewhere: https://www.swebench.com/multimodal.html

[ Parent | on HN ]
Re:A Research Preview of Codex by Snuggly73 on Friday May 16, 2025 @10:35PM (#44011612)
I mean that there is the possibility that swe bench is being specifically targeted for training and the results may not reflect real world performance.
[ Parent | on HN ]