News for hackers. Stuff that matters.
Re:Copilot regurgitating Quake code, including sweary comments by amelius on Friday July 02, 2021 @11:17AM (#27712615)
What are the statistics of Copilot based on a validation set? How often does it get code right?

I want to see hard statistics, not 4 hand-picked examples.

Re:Copilot regurgitating Quake code, including sweary comments by mnky9800n on Friday July 02, 2021 @11:37AM (#27712878)
That's what I thought when I first started working in text generation too. It's highly annoying people pitch their successful models with hand picked examples. It's literally the opposite of STATISTICAL learning imo.
Re:Copilot regurgitating Quake code, including sweary comments by rubatuga on Friday July 02, 2021 @12:29PM (#27713635)
Yeah. And like how would you even devise a metric? Like compile it down to assembly and see if it’s similar logic?
Re:Copilot regurgitating Quake code, including sweary comments by amelius on Friday July 02, 2021 @02:39PM (#27715148)
Well, this is the question which the producers of the tool should answer.

You can't just release a ML tool onto the public if you haven't validated it first.