Re:Risk Assessment of GitHub Copilot
by qbasic_forever on Monday July 12, 2021 @02:49PM
(#27813639)
IMHO they built the opposite of what's actually useful for real-world use. Copilot should have been trained to describe what a selected block of code does, not write a block of code from a description. It could be extremely useful when looking at new or under-documented codebases to have an AI that gives you a rough hint as to what some code might be doing. For example if you select some heinous spaghetti code function, press a button, and get a prompt back that says "This code looks like it's parsing HTML using regex (74.2% confidence)" it could be much easier for folks to be productive on big codebases.
[ Parent
| on HN ]
Re:Risk Assessment of GitHub Copilot
by heavyset_go on Monday July 12, 2021 @06:15PM
(#27815756)
But that would require hiring tons of software engineers to label training data accurately.
Why do that when you can just train a GPT-3 model on public repositories and call it a day?
Re:Risk Assessment of GitHub Copilot
by qbasic_forever on Monday July 12, 2021 @06:40PM
(#27815943)
No presumably copilot skirted that need by just analyzing the AST of code they host and using the nearby comments to identify what a section of code is meant to do. This would use the same dataset but solve the opposite problem, generate a description from a block of code AST as input.
Re:Risk Assessment of GitHub Copilot
by patchguard on Tuesday July 13, 2021 @02:44AM
(#27818195)
> copilot skirted that need by just analyzing the AST of code they host and using the nearby comments to identify what a section of code is meant to do.
I'm curious what it spills out for things like "Todo", or "this is probably broken", etc.
Sorry for adding just noise, but I think this is the most insightful comment I've read on HN this year. Excellent analysis and idea!
Re:Risk Assessment of GitHub Copilot
by moralestapia on Monday July 12, 2021 @06:39PM
(#27815940)
Something like this would be amazing, particularly for poorly written, obfuscated or even disassembled/decompiled code!
Now that is a damn good idea!
it’s a good idea. depending on how “smart” it is it can be extremely hard to pull off
Re:Risk Assessment of GitHub Copilot
by toomuchtodo on Monday July 12, 2021 @03:19PM
(#27813904)
Ideally, you'd train/teach it using PR code reviews. Human labeling and all that jazz.
Re:Risk Assessment of GitHub Copilot
by JetSpiegel on Thursday July 15, 2021 @05:09PM
(#27850168)
> Ideally, you'd train/teach it using PR code reviews.
Which is why, based on Windows state, it will never come out of Microsoft.
Re:Risk Assessment of GitHub Copilot
by randallsquared on Monday July 12, 2021 @06:01PM
(#27815616)
I'm not sure I understand how you envision this working, given the underlying technology. You'd have to have a pretty large cache of such analyses to train on, right?
Re:Risk Assessment of GitHub Copilot
by qbasic_forever on Monday July 12, 2021 @06:41PM
(#27815955)
Github has a huge amount of source code and likely for copilot they already had to transform it into an AST to look at comments and nearby code. This would use the same dataset but build the opposite model--input a block of code AST and get a guess as to what the description (i.e. comment) should be for it.
Re:Risk Assessment of GitHub Copilot
by randallsquared on Monday July 12, 2021 @09:07PM
(#27816779)
My naive assumption is that they don't have nearly that level of control. I'd be surprised if they have an AST step before the tokenizer, or in it.