Re:GitHub releases an ImageNet for code and a CodeSearchNet ...
by jacquesm on Friday September 27, 2019 @09:21AM
(#21091639)
Thank you for responding. So, then the next question is whether or not the code found in public repositories is also representative for the code found in private ones. I can see arguments for it going either way (better, worse), which might have substantial impact on the applicability of the models for code from the class that it never saw.
Re:GitHub releases an ImageNet for code and a CodeSearchNet ...
by mabrocks on Friday September 27, 2019 @09:31AM
(#21091735)
That's an interesting question which we haven't studied so far. There is a good argument to be made that ML models trained on public data do not work well on private data on the interesting queries, where I consider those queries interesting that only make sense in the specific project. The core issue there is around the specialised vocabulary that you would only see in the (private) project, which the model would be unfamiliar with. This could be mitigated by using absurd amounts of data, models that can generalise more easily (character-level/BPE/subtokens/...), and finetuning on project-specific data, but it's an open question nonetheless.
It's also easy to test without access to private repos, by just splitting some repos off as a separate test set, so should hopefully be something that other people can also make progress on!
Re:GitHub releases an ImageNet for code and a CodeSearchNet ...
by jacquesm on Friday September 27, 2019 @10:04AM
(#21092118)
You could test on GH and MS's own private repos to get some idea of the performance. These might still not be representative but it would be an interesting data point.
Re:GitHub releases an ImageNet for code and a CodeSearchNet ...
by throwaway744678 on Friday September 27, 2019 @04:53AM
(#21089807)
What about code from public repository that don't have some kind of open source licence? Won't you be running into a copyright issue somewhere?
Re:GitHub releases an ImageNet for code and a CodeSearchNet ...
by mabrocks on Friday September 27, 2019 @04:55AM
(#21089818)
We only used code from repos that github has marked as using a non-copyleft open source license (i.e., we had an explicit license whitelist and used only repos matching that).
Re:GitHub releases an ImageNet for code and a CodeSearchNet ...
by gmueckl on Friday September 27, 2019 @07:38AM
(#21090745)
So how do you handle repositories that apply different licenses to different subsections of code? E.g. what haopens when an optional helper library is in a subdirectory and licensed differently?
All code is from public repositories. We only used publicly available APIs to obtain the data, so that others can reproduce the results / build on top of the data extraction pipeline we built: https://github.com/github/CodeSearchNet/tree/master/function...