News for hackers. Stuff that matters.
Re:GitHub Copilot Coding Agent by overfeed on Monday May 19, 2025 @02:42PM (#44033267)
> we've merged almost 1,000 pull requests contributed by Copilot

I'm curious to know how many Copilot PRs were not merged and/or required human take-overs.

Re:GitHub Copilot Coding Agent by sethammons on Monday May 19, 2025 @03:29PM (#44033818)
textbook survivorship bias https://en.wikipedia.org/wiki/Survivorship_bias

every bullet hole in that plane is the 1k PRs contributed by copilot. The missing dots, and whole missing planes, are unaccounted for. Ie, "ai ruined my morning"

Re:GitHub Copilot Coding Agent by n2d4 on Monday May 19, 2025 @04:17PM (#44034347)
It's not survivorship bias. Survivorship bias would be if you made any conclusions from the 1000 merged PRs (eg. "90% of all merged PRs did not get reverted"). But simply stating the number of PRs is not that.
Re:GitHub Copilot Coding Agent by tines on Monday May 19, 2025 @06:15PM (#44035527)
As with all good marketing, the conclusions omitted and implied, no?
Re:GitHub Copilot Coding Agent by n2d4 on Monday May 19, 2025 @08:35PM (#44036539)
The implied conclusion ("Copilot made 1000 changes to the codebase") is also not survivorship bias.

By that logic, literally every statement would be survivorship bias.

Re:GitHub Copilot Coding Agent by tines on Monday May 19, 2025 @10:10PM (#44037106)
That’s not the implied conclusion my guy. That’s the statement.
Re:GitHub Copilot Coding Agent by n2d4 on Monday May 19, 2025 @10:34PM (#44037220)
Then what do you claim the implied conclusion is?
Re:GitHub Copilot Coding Agent by Jenk on Tuesday May 20, 2025 @03:20AM (#44038694)
That the number of successful (as in, merged and works) contributions are greater than those that did not.
Re:GitHub Copilot Coding Agent by krainboltgreene on Tuesday May 20, 2025 @02:50AM (#44038526)
Given that Github is continuing with the product and marketing to us it feels sufficient to count that as a conclusion.
Re:GitHub Copilot Coding Agent by MoreQARespect on Monday May 19, 2025 @05:16PM (#44034950)
If they measured that too it would make it harder to justify a MSFT P/E ratio of 29.6.
Re:GitHub Copilot Coding Agent by literalAardvark on Monday May 19, 2025 @04:01PM (#44034169)
"We need to get 1000 PRs merged from Copilot" "But that'll take more time" "Doesn't matter"
Re:GitHub Copilot Coding Agent by worldsayshi on Monday May 19, 2025 @04:08PM (#44034247)
I do agree that some scepticism is due here but how can we tell if we're treading into "moving the goal posts" territory?
Re:GitHub Copilot Coding Agent by overfeed on Monday May 19, 2025 @04:47PM (#44034684)
I'd love to know where you think the starting position of the goal posts was.

Everyone who has used AI coding tools interactively or as agents knows they're unpredictably hit or miss. The old, non-agent Copilot has a dashboard that shows org-wide rejection rates for for paying customers. I'm curious to learn what the equivalent rejection-rate for the agent is for the people who make the thing.

Re:GitHub Copilot Coding Agent by taurath on Monday May 19, 2025 @11:00PM (#44037352)
I think the implied promise of the technology, that it is capable of fundamentally changing organizations relationships with code and software engineering, deserves deep understanding. Companies will be making multi million dollar decisions based on their belief in its efficacy
Re:GitHub Copilot Coding Agent by internet101010 on Tuesday May 20, 2025 @09:23AM (#44041334)
When someone says that the number given is not high enough. I wouldn't consider trying to get an understanding of PR acceptance rate before and after Copilot to be moving the goal posts. Using raw numbers instead of percentages is often done to emphasize a narrative rather than simply inform (e.g. "Dow plummets x points" rather than "Dow lost 1.5%").
Re:GitHub Copilot Coding Agent by Cthulhu_ on Tuesday May 20, 2025 @03:14AM (#44038667)
I feel the same about automated dependency updates, but if your tests and verifications are good, these become trivial.
Re:GitHub Copilot Coding Agent by skydhash on Tuesday May 20, 2025 @10:24AM (#44042028)
Sometimes there are some paradigms shift in the dependency that get past the current tests you have. So it’s always good to read the changelog and plan the update accordingly.
Re:GitHub Copilot Coding Agent by no_wizard on Tuesday May 20, 2025 @02:15PM (#44044314)
Strong automated tests and verifications seem to be nearly as rare as unicorns, at least if you take most of developers feelings on this.

It seems places don't prioritize it, so you don't see it very often. Some developers are outright dismissive of the practice.

Unfortunately, AI won't seemingly help with that

Re:GitHub Copilot Coding Agent by philipwhiuk on Tuesday May 20, 2025 @05:24AM (#44039544)
I'm curious how many were much more than Dependabot changes.
Re:GitHub Copilot Coding Agent by xeromal on Tuesday May 20, 2025 @12:10PM (#44043196)
I see number of PRs as modern LOC, something that doesn't tell me anything about quality.