mabrocks
The problem with all of these things (as with FramaC) is that they only work if you restrict the fragment of programs that you're looking at. Kernel modules work out surprisingly well, because they are not allowed to do a whole lot by convention.
It's also easy to test without access to private repos, by just splitting some repos off as a separate test set, so should hopefully be something that other people can also make progress on!
We also provide a way to evaluate how well your machine learning model works. That's why the blog post says "We’re announcing the CodeSearchNet Challenge evaluation environment."
The released data and code (and hence, the announcement) are meant for data scientists and ML researchers who want to work on this problem, and the rest of the world does not need to care. There are no products or applications of this work at this time. If the terms in the blog post don't mean anything to you, then you are not in the target audience for this announcement, and that's OK.
There are many interesting ideas that you could build on top of this kind of data, and we only scratched the surface so far.
For example, the simple "search" technique we are using as a baseline is based on the idea of joint embeddings: We learn functions f_query and f_js/f_python/... to map from the inputs into some vector space such that, for example, for a (python method, docstring) pair, f_query(docstring) is near to f_python(method). To search given a query, we just do f_query(query) and look for nearest neighbours in all the code we indexed before.
Now, we could also just do f_python(def bubblesort(...): ...) and look for the nearest neighbour that is in C#, and should get out a C# implementation of bubblesort. Similarly, we could apply all kinds of filters on the results (code from highly-starred repos, code that uses framework X, ...) to do more interesting thing.
There wasn't a technical reason (unless you count lazyness as a technical reason) -- we simply had infrastructure for Python specifically lying around from past research projects, which we initially reused.
After we got Nat's feedback, we redid our data processing pipeline completely to be based TreeSitter (which wasn't around when we started thinking about parsing Python), which makes it much easier to scale to the number of programming languages on GitHub.
We used the repo-level license information from github to filter to free non-copyleft licenses, and the license files are actually stored next to the extracted data corpus (see https://github.com/github/CodeSearchNet#Licenses).
All code is from public repositories. We only used publicly available APIs to obtain the data, so that others can reproduce the results / build on top of the data extraction pipeline we built: https://github.com/github/CodeSearchNet/tree/master/function...