News for hackers. Stuff that matters.
Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by mloncode on Thursday September 26, 2019 @06:17PM (#21086167)
Hello folks, this is Hamel from GitHub -- I’m one of the Machine Learning Engineers who worked on this project. The reason we are excited to host this data is that we believe the community will be able to innovate and advance the state of the art much faster if it is provided in a tractable format for machine learning researchers. This data is already public, however there is specialized knowledge required to acquire, parse, dedupe, and clean code from many programming languages at a massive scale on GitHub. We strived to reduce these barriers to encourage greater involvement.

While we present the task of information retrieval as one possible use case of this dataset, we know there could be other practical applications of this data (i.e. code summarization). While we went through great lengths to pre-process the data for the community, the data is still messy and often you will find that there might not be high-quality comments that are aligned with a code snippet we parsed. However, we believe this is part of the excitement of the dataset it poses challenges that machine learning practitioners will have to address.

Code is very different than natural language with regard to its structure and syntactic rules and may benefit from different approaches relative to natural language processing. Our baseline models and benchmarks mostly treat code as natural language, however we are aware that there could be an opportunity to innovate on this front. If anyone creates any interesting projects from this dataset, please do get in touch. Happy to answer any questions!

Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by dchichkov on Friday September 27, 2019 @12:47AM (#21088538)
How would you make sure that any derived works (resulting from the use of this dataset) are properly licensed? It is very likely that this dataset (as per 1909.09436.pdf: "2 million functions, obtained from mechanically scraping and preprocessing associated function documentation") is contaminated by code with dubious licenses, right? And any derived work as a result would also be contaminated, right? What's the plan to deal with this issue?
Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by mabrocks on Friday September 27, 2019 @03:16AM (#21089292)
[I'm one of the Microsoft Research people who worked on this]

We used the repo-level license information from github to filter to free non-copyleft licenses, and the license files are actually stored next to the extracted data corpus (see https://github.com/github/CodeSearchNet#Licenses).

Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by panny on Friday September 27, 2019 @10:19AM (#21092269)
Are you doing anything to help copyleft licensed code repos find infringers? We all know it's common practice for unscrupulous developers and companies to take copyleft code, copy it, and then try to relicensed under permissive licenses on a regular basis. Like Google did with Oracle's GPLed Java source code.

It seems irresponsible for Github to continue distributing infringing code when it clearly has the tools to prevent that happening. Youtube has fingerprinting for infringing music and video. Github should be doing the same with source code.

Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by efdee on Friday September 27, 2019 @11:19AM (#21092916)
> Like Google did with Oracle's GPLed Java source code. That's not what happened -at all-.
Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by bpt3 on Friday September 27, 2019 @11:11AM (#21092818)
How does Github clearly have the tools to prevent copyright infringement from happening?

If you're saying they are the entity in the best position to identify infringement, I would agree, but what tools does Github already have in place to do that?

Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by panny on Friday September 27, 2019 @07:46PM (#21097270)
Github can ID project dependencies and notify devs when one of them is vulnerable to a CVE. They could more easily find licenses for all project dependencies also, they just don't. Github could id private repos using GPL code for non-gpl licensed projects. Github could even automatically apply the correct GPL license and make the projects public, which would satisfy the license.

By failing to do so, Github really seems to be encouraging copyright infringement.

Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by bpt3 on Tuesday October 01, 2019 @12:59PM (#21127711)
Nothing you described here is a trivial problem, nor a problem that Github has any responsibility to unilaterally fix, nor do they have the tools to do so already as you previously stated.

If Github were somehow helping copyright infringers, that would be encouragement. What you're describing is additional effort on their part to actively discourage copyright infringement, which is not the same thing.

Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by desiderantes on Friday September 27, 2019 @10:51AM (#21092610)
>Like Google did with Oracle's GPLed Java source code.

What are you talking about?

Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by panny on Friday September 27, 2019 @07:34PM (#21097217)
Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by glennpratt on Saturday September 28, 2019 @02:58PM (#21102185)
Test files that were not shipped to customers...
Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by ArtWomb on Friday September 27, 2019 @10:01AM (#21092077)
Absolute game changer. We are witnessing the birth of "Big Code"

From the paper, it's not really clear how the resulting binary or intermediate representation fits into the picture? Wouldn't a robust model have to exploit features of the underlying computer architecture as well?

Besides the possible applications mentioned such as recommendation systems and code generation. I am wondering about automatic code annotation and documentation. With the eventual goal of creating a "code tutor" that would assist students in real-time as they type. I think what's interesting about this is that the most talented CS grads tend to have a high correlation with deep understanding not only of the history of digital logic design, but also math, physics etc.

Fantastic work ;)

Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by marmaduke on Friday September 27, 2019 @05:07AM (#21089878)
thanks for the dataset release! So many questions. Have you thought about any approaches to assess quality/accuracy of comments by using the data itself? One could assume a distribution of accuracy and look for correlated metrics on the comment? Other information such as author or project could be helpful if present, by simplying regularizing comment quality over author/project? If GitHub would be a primary beneficiary of any techniques developed with this dataset, would they volunteer any compute resources toward running models?
Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by mallamanis on Friday September 27, 2019 @08:38AM (#21091215)
[I'm one of the Microsoft Research people who worked on this]

Thanks for your questions! We have thought of many heuristics but we didn't want to constrain the dataset release on some heuristic that we picked, possibly ruining the dataset. Participants in the challenge should feel free to apply additional filters as they see fit. For example, this [1] work could be useful as a filtering method.

Unfortunately, we do not have the budget to provide any compute resources to help with running the models at this time. Note that any techniques developed with this dataset will be owned by those who develop them and it's up to them how/if they will make them available/open-source.

[1] https://arxiv.org/abs/1806.04616

Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by jacquesm on Friday September 27, 2019 @02:43AM (#21089115)
Have you used code from private repositories in the training of this model?
Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by mabrocks on Friday September 27, 2019 @03:12AM (#21089272)
[I'm one of the Microsoft Research people who worked on this]

All code is from public repositories. We only used publicly available APIs to obtain the data, so that others can reproduce the results / build on top of the data extraction pipeline we built: https://github.com/github/CodeSearchNet/tree/master/function...

Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by jacquesm on Friday September 27, 2019 @09:21AM (#21091639)
Thank you for responding. So, then the next question is whether or not the code found in public repositories is also representative for the code found in private ones. I can see arguments for it going either way (better, worse), which might have substantial impact on the applicability of the models for code from the class that it never saw.
Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by mabrocks on Friday September 27, 2019 @09:31AM (#21091735)
That's an interesting question which we haven't studied so far. There is a good argument to be made that ML models trained on public data do not work well on private data on the interesting queries, where I consider those queries interesting that only make sense in the specific project. The core issue there is around the specialised vocabulary that you would only see in the (private) project, which the model would be unfamiliar with. This could be mitigated by using absurd amounts of data, models that can generalise more easily (character-level/BPE/subtokens/...), and finetuning on project-specific data, but it's an open question nonetheless.

It's also easy to test without access to private repos, by just splitting some repos off as a separate test set, so should hopefully be something that other people can also make progress on!

Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by jacquesm on Friday September 27, 2019 @10:04AM (#21092118)
You could test on GH and MS's own private repos to get some idea of the performance. These might still not be representative but it would be an interesting data point.
Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by throwaway744678 on Friday September 27, 2019 @04:53AM (#21089807)
What about code from public repository that don't have some kind of open source licence? Won't you be running into a copyright issue somewhere?
Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by mabrocks on Friday September 27, 2019 @04:55AM (#21089818)
We only used code from repos that github has marked as using a non-copyleft open source license (i.e., we had an explicit license whitelist and used only repos matching that).
Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by gmueckl on Friday September 27, 2019 @07:38AM (#21090745)
So how do you handle repositories that apply different licenses to different subsections of code? E.g. what haopens when an optional helper library is in a subdirectory and licensed differently?
Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by airocker on Thursday September 26, 2019 @11:01PM (#21088031)
Could you have code reviews in the data?
Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by mloncode on Thursday September 26, 2019 @11:15PM (#21088083)
The unit of observation for this data are functions or methods and their associated docstrings or top-level comments. So there are no code reviews in the data. However, we do include metadata including the SHA and owner/repo which would allow you to retrieve this information! What were you thinking of doing with code reviews?
Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by airocker on Friday September 27, 2019 @12:25PM (#21093653)
We can try to write an automatic code review system?
[dead]
Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by kstenerud on Friday September 27, 2019 @05:33AM (#21090018)
If you had met mloncode at a party, would you have said it this way to his face after he said "Hey, check out this thing I helped build to make code search better"?

Do you realize how rude and disrespectful that tone is?

There is a civil way to air one's views; one that we use naturally in face-to-face conversations. Yet once it's from behind a computer screen, we somehow seem to think it's okay to lash out in a barrage of hurtful language...

Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by zamfi on Friday September 27, 2019 @03:44AM (#21089454)
Seriously, how dare someone at GitHub release a large dataset that might be interesting while their website’s basic search functionality is still terrible!? Why isn’t everyone at GitHub working to solve just this problem??

C’mon, cut these folks some slack — GitHub is huge, and one annoying adjacent feature doesn’t give you the right to attack these folks like this.

Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by mattigames on Friday September 27, 2019 @04:15AM (#21089603)
This is not an "attack", is just straight forward feedback, and really one I agree with, a company that has received more than 100 millions on investment shouldn't have such a bad search feature
Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by zamfi on Friday September 27, 2019 @04:59AM (#21089839)
Really? Did you read the same post I did?

> Just get the stupid dedupping done, for God's sake.

> You have no excuse to so badly that a teenager can beat you by simple handcrafted rules.

> First learn and do the basics and then come here to talk about your need for ML.

> No one needs to suffer like this.

Let’s see, the OP belittles the original authors for something they’re not responsible for, implies that they’re incompetent, tells them to go away until their problem is fixed, and, oh, for god measure: invokes a deity.

This is not “straight forward feedback” to me.

How about we just cut all the insults and shift the tone a bit:

> GitHub has a really poor search experience compared with most modern web services. Search doesn't need fancy ML or DL, and dedupping alone would help a ton. A simple TF-IDF similarity metric would make GitHub’s search 10,000% better. You even know forks, names of files, folders, number of stars.

> PS: Sorry, I had to get this out. I've spent better part of my productivity in past two years shifting through page number 50 and 100 of undedupped results that GitHub returns and it’s been extremely frustrating.

See how this doesn’t blame the authors for something that isn’t their fault? Doesn’t imply they’re stupid or incompetent for some other feature they have nothing to do with? Doesn’t tell them to go away? And a bonus: It presents some approaches to solve a problem the author is frustrated by.

My experience ranting on the internet is that personal attacks don’t lead to feature improvements.

Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by forkerenok on Friday September 27, 2019 @07:43AM (#21090772)
I didn't feel that the commenter attacked/belittled anyone personally. The comment seemed to address the company (albeit in an overly emotional way).

I think the pointed out asymmetry was spot on.

Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by Keloo on Friday September 27, 2019 @05:14AM (#21089911)
I think the OP intentionally wrote it very aggressive, he even put the <rant> tag to be specific.
Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by littlestymaar on Friday September 27, 2019 @05:49AM (#21090094)
While I agree with OP's sentiment (gh's search is surprisingly poor), putting a warning “I'm going to be a jerk” isn't an excuse to be one…
Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by rob74 on Friday September 27, 2019 @10:13AM (#21092218)
While I probably would have disagreed with the OP's tone (I can't see the post anymore because of too many downvotes, guess I'm late to the party), I agree in principle: GitHub could have drastically improved their search function already years ago with some pretty simple ideas (low-hanging fruit and such), no need to throw ML at it...
Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by imron on Friday September 27, 2019 @10:21AM (#21092289)
Case sensitive search is all I want.
Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by hombre_fatal on Friday September 27, 2019 @05:34AM (#21090024)
I downvoted this because ranty tantrums don't belong on HN.
Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by Der_Einzige on Friday September 27, 2019 @06:53AM (#21090474)
Lol, GitHub search is far better than tf-idf you are seriously mistaken if you think that tf-idf will be better
Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by avip on Friday September 27, 2019 @06:59AM (#21090512)
Downvoted for tone and non-constructive, unsubstantiated criticism.
Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by tallanvor on Friday September 27, 2019 @06:10AM (#21090211)
Why are you hosting the data in AWS instead of on Azure?
Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by flohofwoe on Friday September 27, 2019 @05:50AM (#21090100)
I know this sounds silly because code on github is visible to everybody anyway and that's a good thing, but I would appreciate a way to opt out my own code from such automated data gathering programs for machine learning purposes.

Something like a robots.txt for github projects. Not that anybody would really care, only to make my intent clear that I don't support this sort of mass data gathering nonsense.

Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by kogir on Friday September 27, 2019 @07:20AM (#21090630)
You opt out by using a restrictive license or making your repo private.
Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by flohofwoe on Friday September 27, 2019 @10:26AM (#21092340)
Using a restrictive license seems a bit extreme when the whole point of a public github repo is to make the code available for other programmers.

I consider this mass data gathering use for machine learning an edge case which should not harm the 'legitimate' users of the repository.

Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by bpt3 on Friday September 27, 2019 @11:13AM (#21092849)
Why do you see this effort as not "legitimate"?
Re:GitHub releases an ImageNet for code and a CodeSearchNet ... by zelly on Friday September 27, 2019 @01:23PM (#21094163)
Self-host your own Git browser. Gitlab or Cgit is still good which is still used by the biggest project in the world (linux kernel).