AutoCodeRover resolves 22% of real-world GitHub issues in SWE-bench lite

Re:AutoCodeRover resolves 22% of real-world GitHub issues in... by joenot443 on Tuesday April 09, 2024 @08:39AM (#39978769)

This is super fascinating stuff, excellent work. As most of us don't have the time to read the entirety of the paper, are you able to directly link to some issues which have been landed and closed? Some personal favorites would be awesome :)

I think I speak for others when I say the best way to judge the efficacy of this project is some real-world, on-site examples of it being used in prod. I'm especially curious for its performance in feature-request or flakey bug report type issues as opposed to reliable test failures. I expect the former is much tougher!

[ Parent | on HN ]

Re:AutoCodeRover resolves 22% of real-world GitHub issues in... by yuntong on Tuesday April 09, 2024 @10:00AM (#39979530)

Thank you for your interest. There are some interesting examples in the SWE-bench-lite benchmark which are resolved by AutoCodeRover:

- From sympy: https://github.com/sympy/sympy/issues/13643. AutoCodeRover's patch for it: https://github.com/nus-apr/auto-code-rover/blob/main/results...

- Another one from scikit-learn: https://github.com/scikit-learn/scikit-learn/issues/13070. AutoCodeRover's patch (https://github.com/nus-apr/auto-code-rover/blob/main/results...) modified a few lines below (compared to the developer patch) and wrote a different comment.

There are more examples in the results directory (https://github.com/nus-apr/auto-code-rover/tree/main/results).

[ Parent | on HN ]

Re:AutoCodeRover resolves 22% of real-world GitHub issues in... by dboreham on Tuesday April 09, 2024 @09:51AM (#39979440)

fwiw the example issue highlighted in the post was already fixed by a human 3 years ago so I wouldn't expect to see much in the way of real life fixed issues yet.

[ Parent | on HN ]