AutoCodeRover resolves 22% of real-world GitHub issues in SWE-bench lite

Re:AutoCodeRover resolves 22% of real-world GitHub issues in... by withinboredom on Tuesday April 09, 2024 @08:16AM (#39978599)

I would be interested to see how it performs on end-user software where bug reports are nebulous at best, ridiculous at worst. Furthermore, most of those fixes tend to be upstream bugs and very rarely anything to do with the actual software.

[ Parent | on HN ]

Re:AutoCodeRover resolves 22% of real-world GitHub issues in... by abhikrc on Tuesday April 09, 2024 @08:33AM (#39978722)

The entire setup is available for inspection, please see

https://github.com/nus-apr/auto-code-rover

if you need example bugs we can provide that too. Some examples also appear in the arxiv paper, please see

https://arxiv.org/abs/2404.05427

[ Parent | on HN ]

Re:AutoCodeRover resolves 22% of real-world GitHub issues in... by joenot443 on Tuesday April 09, 2024 @08:39AM (#39978769)

This is super fascinating stuff, excellent work. As most of us don't have the time to read the entirety of the paper, are you able to directly link to some issues which have been landed and closed? Some personal favorites would be awesome :)

I think I speak for others when I say the best way to judge the efficacy of this project is some real-world, on-site examples of it being used in prod. I'm especially curious for its performance in feature-request or flakey bug report type issues as opposed to reliable test failures. I expect the former is much tougher!

[ Parent | on HN ]

Re:AutoCodeRover resolves 22% of real-world GitHub issues in... by yuntong on Tuesday April 09, 2024 @10:00AM (#39979530)

Thank you for your interest. There are some interesting examples in the SWE-bench-lite benchmark which are resolved by AutoCodeRover:

- From sympy: https://github.com/sympy/sympy/issues/13643. AutoCodeRover's patch for it: https://github.com/nus-apr/auto-code-rover/blob/main/results...

- Another one from scikit-learn: https://github.com/scikit-learn/scikit-learn/issues/13070. AutoCodeRover's patch (https://github.com/nus-apr/auto-code-rover/blob/main/results...) modified a few lines below (compared to the developer patch) and wrote a different comment.

There are more examples in the results directory (https://github.com/nus-apr/auto-code-rover/tree/main/results).

[ Parent | on HN ]

Re:AutoCodeRover resolves 22% of real-world GitHub issues in... by dboreham on Tuesday April 09, 2024 @09:51AM (#39979440)

fwiw the example issue highlighted in the post was already fixed by a human 3 years ago so I wouldn't expect to see much in the way of real life fixed issues yet.

[ Parent | on HN ]