Re:GitHub releases an ImageNet for code and a CodeSearchNet ...
by jacquesm on Friday September 27, 2019 @09:21AM
(#21091639)
Thank you for responding. So, then the next question is whether or not the code found in public repositories is also representative for the code found in private ones. I can see arguments for it going either way (better, worse), which might have substantial impact on the applicability of the models for code from the class that it never saw.
[ Parent
| on HN ]
Re:GitHub releases an ImageNet for code and a CodeSearchNet ...
by mabrocks on Friday September 27, 2019 @09:31AM
(#21091735)
That's an interesting question which we haven't studied so far. There is a good argument to be made that ML models trained on public data do not work well on private data on the interesting queries, where I consider those queries interesting that only make sense in the specific project. The core issue there is around the specialised vocabulary that you would only see in the (private) project, which the model would be unfamiliar with. This could be mitigated by using absurd amounts of data, models that can generalise more easily (character-level/BPE/subtokens/...), and finetuning on project-specific data, but it's an open question nonetheless.
It's also easy to test without access to private repos, by just splitting some repos off as a separate test set, so should hopefully be something that other people can also make progress on!