News for hackers. Stuff that matters.
Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by mabrocks on Friday September 27, 2019 @04:16AM (#21089612)
[I'm one of the Microsoft Research people who worked on this]

There wasn't a technical reason (unless you count lazyness as a technical reason) -- we simply had infrastructure for Python specifically lying around from past research projects, which we initially reused.

After we got Nat's feedback, we redid our data processing pipeline completely to be based TreeSitter (which wasn't around when we started thinking about parsing Python), which makes it much easier to scale to the number of programming languages on GitHub.

Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by Dowwie on Friday September 27, 2019 @05:40AM (#21090048)
In other words, the system is ready to be trained for languages not on the initial list, without requiring additional development?
Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by mabrocks on Friday September 27, 2019 @06:33AM (#21090355)
You would need to extend the data-processing pipeline for the new language, which in the best case only requires to adapt the standard wrapper around the Tree Sitter parser. The wrapper needs to take care of language-specific details (e.g., where to find the documentation for a function, what methods should be filtered out, etc.) but can be fairly small. See https://github.com/github/CodeSearchNet/blob/master/function... for examples.