Re:AutoCodeRover resolves 22% of real-world GitHub issues in...
by skywhopper on Tuesday April 09, 2024 @10:21AM
(#39979722)
SWE-bench Lite is a subset of extremely simple issues from a cherry-picked subset (SWE-bench) of a handful of large (presumably well-run) Python-only projects.
Here are some rules they used to trim down the SWE-bench Lite problems:
* We remove instances with images, external hyperlinks, references to specific commit shas and references to other pull requests or issues.
* We remove instances that have fewer than 40 words in the problem statement.
* We remove instances that edit more than 1 file.
* We remove instances where the gold patch has more than 3 edit hunks (see patch).
Here are some rules they used to trim down the SWE-bench Lite problems:
* We remove instances with images, external hyperlinks, references to specific commit shas and references to other pull requests or issues.
* We remove instances that have fewer than 40 words in the problem statement.
* We remove instances that edit more than 1 file.
* We remove instances where the gold patch has more than 3 edit hunks (see patch).
See https://www.swebench.com/lite.html