News for hackers. Stuff that matters.
Re:Differences between the word2vec paper and its implementa... by throwaway287391 on Monday June 03, 2019 @07:26PM (#20090058)
This sort of thing is aggravating to read. Frankly it comes off as really entitled. As researchers, the expectation is now that we not only have to do the research and write a paper like the good old days, but we have to release the code too. Okay, fine. But now that's not enough either -- the code has to be well-documented and clean. Ugh, alright, fine -- it's going to take me a few extra weeks of not doing research, but I'll clean up all the code, rerun experiments to make sure it all still works like before, and add a bunch of documentation. But no, still not good enough -- it has to run at the press of a button in your particular programming environment. If we don't know how to write a script (or couldn't be bothered to spend the time writing one) to check that the data is on disk and, if not, crawl a website to download some huge dataset in one click, test our code on your OS, your CPU/GPU/TPU/..., etc., we were being "insincere" with our open-sourcing efforts.
Re:Differences between the word2vec paper and its implementa... by KirinDave on Monday June 03, 2019 @08:58PM (#20090717)
Pardon me, maybe I just misunderstood the whole idea of research but what good is it if it's not reproducible?

I can understand it may be part of a meaningful personal journey for you, and I appreciate that. But if no one else can validate your research they're correct to discredit it and you.

So what is the optimal outcome here? Should we hold you to a standard of reproducibility even if it is as minimal as, "actually describe your algorithms correctly and don't misrepresent a piece of code and a paper?" Or should everyone just decide you can find your own research funding if it's not going to help anyone?

Re:Differences between the word2vec paper and its implementa... by archgoon on Monday June 03, 2019 @09:37PM (#20090931)
The original idea behind "reproducible" is that the ideas conveyed in the paper should be enough to reproduce the results. Physicists and biologists are not expected to drive over to your lab to figure out what's wrong with your setup.

Now, that said, reproducibility is terrible in many fields. CS has an opportunity to act as a trailblazer here, but it should be noted that this would be holding themselves to a higher standard than their peers in other fields. As a result, there's going to be a learning process for everyone as they figure out how to make this all work. :)

Re:Differences between the word2vec paper and its implementa... by throwawayjava on Monday June 03, 2019 @10:20PM (#20091175)
Some pretty good computer science got done before devops was gifted to the world.

And some pretty good science got done before computer scientists were gifted to the world.

I'm genuinely skeptical that modern software engineering practices are a good way of thinking about reproduction in science. Even in computer science. There's a lot that scientists can learn from software engineering (and in fact I've helped run workshops in the past on exactly this topic), but science is not engineering.

Re:Differences between the word2vec paper and its implementa... by KirinDave on Wednesday June 05, 2019 @10:51AM (#20104983)
> Some pretty good computer science got done before devops was gifted to the world.

I'm happy to talk about this if you want. One of the most important aspects of this work was that people like Dijkstra started using notions that approached what real computers could read while remaining human-readable. This is some measure of classical "reproducibility". And work like McCarthy's was revolutionary in part because it was a definition of reproducibility as a result!

I can give examples of shockingly good papers that are struggling to see the light of day in their industry because they're written in ways that make them hard not only to understand, but to reproduce.

So don't presume to lecture me about this. Part of the reason the word2vec paper stands out is precisely because this is such a deviation from the norm to have a paper misrepresent its most fundamental component: the algorithm.

Re:Differences between the word2vec paper and its implementa... by KirinDave on Wednesday June 05, 2019 @10:35AM (#20104798)
Okay, by the standard you laid out: the paper that is the subject of this paper failed.

But similarly, if you say, "Here are our statistical models, they're in a notation for a private, custom MCMC system you can't use" that's probably failing the standard even if the work is good.

Re:Differences between the word2vec paper and its implementa... by seanmcdirmid on Monday June 03, 2019 @09:28PM (#20090884)
If I have to go dig up someone’s code to reproduce their result because I don’t believe/understand the idea in its published form, they’ve already lost me.

Word2vec is a great example of a piece of research that conveys a great idea with plausible results, where in fact, the numbers are less useful than the story behind them. Of course, google is a big corp that can just put out useful interesting stuff without worrying about how to play the science game.

Re:Differences between the word2vec paper and its implementa... by p1esk on Monday June 03, 2019 @09:23PM (#20090857)
Pardon me, maybe I just misunderstood the whole idea of "reproducible" but it is only not reproducible if you tried to implement the ideas described in the paper yourself, and I mean really tried, and contacted the authors for help, and still failed to achieve the claimed results.
Re:Differences between the word2vec paper and its implementa... by throwawayjava on Monday June 03, 2019 @09:47PM (#20090995)
> if you tried to implement the ideas described in the paper yourself, and I mean really tried, and contacted the authors for help, and still failed to achieve the claimed results.

This is the actually important definition of "reproducible".

"has an install.sh" is really nice, but it's far more important that an informed reader can recreate the artifact for themselves from the written description.

The "must be able to apt install or it's not real science" is a particularly dangerous path to go down. It's how you end up with e.g. shit loads of well-engineered LISP or FORTRAN code with no actual scientific insights or knowledge transfer. Which has actually happened in the past. A lot.

Re:Differences between the word2vec paper and its implementa... by telchar on Tuesday June 04, 2019 @11:06PM (#20101153)
I don't think people in this thread are insisting on a downloadable package. Just that the algorithm published in the write-up accurately describe the work that was actually done to achieve the stated results. If even that threshold can't be met, IMO the authors are polluting the literature with bad work.
Re:Differences between the word2vec paper and its implementa... by derefr on Monday June 03, 2019 @11:41PM (#20091648)
> and contacted the authors for help

Isn’t the whole point of a paper, to obviate needing to talk to the authors? Like, so that science is a ratchet that doesn’t slip backward the moment the authors die?

Re:Differences between the word2vec paper and its implementa... by visarga on Tuesday June 04, 2019 @01:59AM (#20092306)
Related to w2v, the proof is in the vectors. They work. And, by the way, w2v has had a whole family of variants which were built on top of the original implementation (such as Doc2Vec, Doc2VecC and others).

Related to most ML code, it's not written in C but in Python frameworks that implement whole layers as building blocks. So it's not as hard to read/change. You can take something complex like the transformer and understand it in a few minutes. The most puzzling part with framework code is figuring out the shape of tensors and, sometimes, what each dimension means.

Regarding reproducibility - it's hard to achieve on account of parallelisation. You see, neural nets use floats, and floats are not real numbers. Half the float numbers lie between [-1, 1] and the rest outside. If you do 1e-10+1e10 you get 1e10. So if you add 1000 times 1e-10 to 1e+10 you still get 1e+10, but if you add them up together first, the result is different. So float summation is not commutative. Depending on race conditions the order in which summation occurs could change and the result change as well, even if you use the same random seeds. And if we give up parallelism then we can't run the experiments any more.

Re:Differences between the word2vec paper and its implementa... by Dylan16807 on Tuesday June 04, 2019 @07:30AM (#20093921)
For almost all ML training, it's straightforward to break it into a fixed number of chunks without race conditions.

That's if float rounding issues are even a big problem in the first place. If your results are within .1% over a few runs it's reproducible enough for most purposes.

Re:Differences between the word2vec paper and its implementa... by throwawayjava on Monday June 03, 2019 @09:44PM (#20090977)
> what good is it if it's not reproducible?

I am one of those people who actually did the extra weeks/months to properly test/review/document/release my code and data sets (you can apt-get install my "research artifacts").

In retrospect, it was a poor use of my time and a poor use of my sponsoring institution's time. "apt-get install" is NOT what we mean by "reproducible" in science.

High-quality or easy to install code is not necessary for a result to be reproducible. True reproduction would mean coding the algorithm from scratch by following the written description, and that's what it literally means in most other fields of science.

You can't download & install a Large Hadron Collider in an afternoon. Does that mean the LHC experiments are "not reproducible"? Of course not.

But that's not even the important point. The really important point is that, in most cases, high-quality code is not even sufficient for a result to be reproducible! See: the article we're discussing.

IMO, the "blindly rerun the code" definition of reproduction is actually a HUGE barrier to creating a true culture of reproducability in computer science. It results in super lazy reviewing where "public source code that's easy to install and puts the correct-sounding shit into STDOUT" becomes a stand-in for "paper actually describes a novel idea in enough detail that it can be truly reproduced".

> So what is the optimal outcome here?

An optimal allocation of scientists' time and effort.

As a scientist who has actually done that leg work, I don't think packaging code so that it runs with a single click is the best use of public money in science in 99.9999% of cases. That time is much better spent on writing and other dissemination explaining the ideas that make the code work (in some cases well-documented source code is the best description but in other cases prose is much more effective and illuminating). Or on coming up with new ideas that are even better than the old ones.

Which I guess is just another way of saying that scientists should spend their time on science, not engineering.

P.S. When shitting on "scientists" for not being good enough software engineers, please remember who's going to be doing the actual work you're demanding. It's mostly phd students who make $30K/yr. And they have to do this work in their free time because their 60 hr/wk day job is fully allocated to doing the actual science. I.e., treat scientists who maintain their code as you would treat FOSS contributors who are making 5x-10x+ less than you while working longer hours. Because maintaining high-quality code is something they are almost certainly doing in their free time.

Re:Differences between the word2vec paper and its implementa... by KirinDave on Wednesday June 05, 2019 @10:46AM (#20104929)
> High-quality or easy to install code is not necessary for a result to be reproducible. True reproduction would mean coding the algorithm from scratch by following the written description, and that's what it literally means in most other fields of science.

Which failed here. However, you're going to have a difficult time convincing many people that the attributes you described would be bad properties, just that they might not justify their expense.

> You can't download & install a Large Hadron Collider in an afternoon. Does that mean the LHC experiments are "not reproducible"? Of course not.

By the same token though, the LHC repeats experiments and solicits feedback on how to improve their methods, which they go to great lengths to publish and simulate, because they're aware of this problem.

> IMO, the "blindly rerun the code" definition of reproduction is actually a HUGE barrier to creating a true culture of reproducability in computer science. It results in super lazy reviewing where "public source code that's easy to install and puts the correct-sounding shit into STDOUT" becomes a stand-in for "paper actually describes a novel idea in enough detail that it can be truly reproduced".

Ah, yes. Yes. "If this code is TOO reproducible then people might reproduce it, and handwave handwave the quality of papers would decline.

That's certainly NOT the case in pure CS papers, which have only improved since the days when folks felt that "Lenses, Bananas and Barbed Wire" was how folks should go about writing papers.

Now, physics might be different. But there is surely a middle ground between, "I've shipped you a LHC just plug in in lol" and "This paper doesn't even remotely describe how we achived the results."

If you believe that wasting the time of scientists is bad, then surely you're for clear papers with accurate descriptions of the methods so that those who go and reproduce your work are not sent on wild goose chases?

> As a scientist who has actually done that leg work, I don't think packaging code so that it runs with a single click is the best use of public money in science in 99.9999% of cases.

No, we got that part. But surely someone does and maybe you can design your work to leverage that rather than reproducing and discarding scaffolding. My big concern here is that a lot of scientists (like you claim to be) are underqualified and unpracticed at software, and thus are surely seeing at least some aspect of their work distorted by software and hardware issues.

> Which I guess is just another way of saying that scientists should spend their time on science, not engineering.

Scientists are not going to be able to escape engineering. No one else is going to build what they need besides them.

> please remember who's going to be doing the actual work you're demanding. It's mostly phd students who make $30K/yr. And they have to do this work in their free time because their 60 hr/wk day job is fully allocated to doing the actual science.

Yeah, I'm aware. I suspect their lot would be better if your attitude wasn't that their work is disposable and unimportant.

Re:Differences between the word2vec paper and its implementa... by patrick5415 on Tuesday June 04, 2019 @03:21AM (#20092669)
Reproducible research does not mean you don’t have to do any work. The original author isn’t going to come to your lab and clean your pipettes nor should they. Grow up.
Re:Differences between the word2vec paper and its implementa... by cfcf14 on Tuesday June 04, 2019 @05:36AM (#20093389)
The research published in this case (the original word2vec paper) is quite literally not what it claims to be. It is factually incorrect. Is that really the responsibility of the individual attempting to reproduce the work to identify and correct?

How wrong should the original work be before it ceases to be a case of 'clean your own pipettes'?

Re:Differences between the word2vec paper and its implementa... by patrick5415 on Tuesday June 04, 2019 @12:49PM (#20096809)
I’d say at the same point that others in the field would generally agree it’s fradulent. I’m not familiar enough with the field of word2vec to say if this is the case or not. But regardless, asking that the implementation be representative of the paper is a lot different than demanding docker containers and install scripts.
Re:Differences between the word2vec paper and its implementa... by KirinDave on Wednesday June 05, 2019 @10:48AM (#20104948)
Patrick,

Could you PLEASE consider reading the article we're all discussing before you roll into the comments section of an article about it with your strong-but-loosely-held opinions? You're arguing against a point that almost no one is putting forward (that software should be one click).

Re:Differences between the word2vec paper and its implementa... by liquidify on Monday June 03, 2019 @07:42PM (#20090221)
If there are bugs in your code that would have prevented you from being able to do what you claim to do, then to the readers of your paper, it seems more likely that you might not have performed the experimentation that you claim to.
Re:Differences between the word2vec paper and its implementa... by throwaway287391 on Monday June 03, 2019 @07:50PM (#20090273)
What? This has nothing to do with bugs in the code that would've affected the original author's ability to run the original experiments described in the paper. The user I replied to is talking about things like hard-coded data paths and wanting to be able to run the code at the press of a button on any platform. The expectation for this level of cross-platform compatibility is difficult to achieve for engineering firms with millions to invest in making it work properly. He's mad that individual machine learning researchers don't always do a great job of making it happen for their cutting-edge research which was a difficult enough engineering task to get running on their machine.
Re:Differences between the word2vec paper and its implementa... by gcommer on Monday June 03, 2019 @07:59PM (#20090344)
Reread the original post:

> I have to fix bugs before the code can run. I'm very curious how did the author run that code with the bugs.

Re:Differences between the word2vec paper and its implementa... by throwaway287391 on Monday June 03, 2019 @08:17PM (#20090473)
You're right, I misread that part.
[dead]
Re:Differences between the word2vec paper and its implementa... by dang on Tuesday June 04, 2019 @12:15AM (#20091840)
Personal attacks will get you banned here. Would you please review the site guidelines and not post like this again?

https://news.ycombinator.com/newsguidelines.html

Re:Differences between the word2vec paper and its implementa... by jonas21 on Monday June 03, 2019 @11:00PM (#20091368)
FWIW, I downvoted because because you somehow managed to break about half of the HN comment guidelines [1] in a single comment, in particular:

* Be kind. Don't be snarky. Comments should get more thoughtful and substantive, not less, as a topic gets more divisive.

* When disagreeing, please reply to the argument instead of calling names. "That is idiotic; 1 + 1 is 2, not 3" can be shortened to "1 + 1 is 2, not 3."

* Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith.

* Please don't make insinuations about astroturfing. It degrades discussion and is usually mistaken. If you're worried, email us and we'll look at the data.

* Please don't comment about the voting on comments. It never does any good, and it makes boring reading.

[1] https://news.ycombinator.com/newsguidelines.html

Re:Differences between the word2vec paper and its implementa... by throwaway287391 on Monday June 03, 2019 @09:49PM (#20091005)
I don't have much to say to most of this, but I'll point out that I didn't and couldn't have downvoted you as none of my throwaways have >=500 karma or whatever the threshold is.
[dead]
Re:Differences between the word2vec paper and its implementa... by throwawayjava on Monday June 03, 2019 @10:46PM (#20091298)
Man, I hate when I get dragged into these sorts of intense emotional downs by silly internet conversations with strangers.

FWIW I undowned your posts; internet karma is about communication/moderation, and sometimes has the opposite of the intended effect. I've been there.

Anyways, find some time to pet a dog/cat or fly a kite today.

Re:Differences between the word2vec paper and its implementa... by p1esk on Monday June 03, 2019 @09:38PM (#20090942)
Typically when I post code for my papers, I try to clean it up, because it's an ugly mess that is hard to understand (even myself sometimes). This clean up process might introduce bugs. I'm not going to thoroughly test everything, I have next paper deadline coming up and my adviser could not care less about my code. But, if you're interested in my research and having trouble reproducing my results, email me and I'll help you.
Re:Differences between the word2vec paper and its implementa... by throwawayjava on Monday June 03, 2019 @10:50PM (#20091318)
...and university/company sometimes makes you wipe out the git commit log, so you can't always just use "HEAD at time of submission".

Pro tip: at paper submission time, md5sum your code and also git tag it in your private repo. When you release the code, if you have to / want to release with a clean history, make submission-time your initial commit and then make the current state of the repo your second commit. I've never encountered an institution that won't allow that level of history in the code release, even places that are pretty hard core about wiping pre-release history.

Re:Differences between the word2vec paper and its implementa... by caenorst on Monday June 03, 2019 @11:01PM (#20091371)
And don't you think it would have been more efficient to write it down clean at first with good comments and code practice ? Those good practices are here for a reason and not just for annoying you for the sack of "beauty". You would save more time for yourself when you wanna expand the paper further or just reuse some part of it, have better reputation for your work, and gain some citation because people will have the tendency to use your code as a starting point.
Re:Differences between the word2vec paper and its implementa... by inimino on Monday June 03, 2019 @11:21PM (#20091519)
"Writing it down clean at first" implies that you know what is going to work when you start. If that's the case, it's not research.
Re:Differences between the word2vec paper and its implementa... by patrick5415 on Tuesday June 04, 2019 @03:39AM (#20092779)
No, it would not be more efficient. I’ve tried this route and it hands down consumes way more time. Until it’s clear that something will get re-used, it’s most efficient to leave it dirty.

The software types love to sht on researchers for their poor quality code. Yet a few threads over, there’s always a discussion about how more tests don’t make it onto the kanban board (or whatever you guys are using these days) when crunch time approaches. It’s not just us.

Re:Differences between the word2vec paper and its implementa... by randcraw on Monday June 03, 2019 @07:57PM (#20090330)
Agreed. If your results are not reproducible by others, the publication should issue a retraction and withdraw your work. Code that works is essential.
Re:Differences between the word2vec paper and its implementa... by dodobirdlord on Monday June 03, 2019 @11:59PM (#20091755)
Nonsense! The "reproducible" in "reproducible result" doesn't mean anything except that what is described in the paper can be reproduced. Code that works is nice and all, but totally irrelevant to the question of reproducibility. Would you claim that that the Higgs paper must be retracted because CERN won't ship you the LHC? They've got ATLAS and CMS to demonstrate reproducibility, and they're wildly different!
Re:Differences between the word2vec paper and its implementa... by Volt on Monday June 03, 2019 @10:54PM (#20091344)
I hope you're a scientist.
Re:Differences between the word2vec paper and its implementa... by Veedrac on Monday June 03, 2019 @07:48PM (#20090264)
If it was just one thing or the other, perhaps he wouldn't have been so critical of it. People are going to understand if you require a few paths to be changed or some files to be downloaded manually into specific relative locations. The ask is only that somebody should be able to use the code without doing their own novel research, and be able to verify it's doing what it is claimed to be doing. If this means your download links need to point to the right location, the code has to run in a location other than /home/hellokitty/ml, and that it needs to meet standards of readability—well, good, that's what the scientific process is about.
Re:Differences between the word2vec paper and its implementa... by caenorst on Monday June 03, 2019 @07:58PM (#20090338)
I have spent enough time to debunk papers to know that a lot of researchers are shamelessly cheating or make mistakes (as EVERYBODY does). Reviewing your code should be part of the peer review process.

Sorry but I don't trust your research just based on yours words and cherry picked images.

Also research is incremental, so producing a proper code on which other can work on top of should be part of the CONTRIBUTION.

Re:Differences between the word2vec paper and its implementa... by ur-whale on Monday June 03, 2019 @09:52PM (#20091020)
I fully agree, and would actually go a lot further.

A very large number of research papers I've read in my life were so poorly written as to being near impossible to understand.

The time it took to infer the actual meaning of certain parts of certain papers was vastly larger than just reading the code would have been.

It has become very clear to me that being a strong CS researcher does not necessarily imply mastery of language or even the very basic ability to explain one's ideas. As a matter of fact, with very few exception (eg Feynmann), the two skills seem to be at odds with one another.

On the other hand, code can be read and understood much more readily, especially when it can be experimented with, if only to insert printfs in it to understand what it does.

The culture of scientific publication in CS must change. The "I'm pressured by my advisor and therefore have no time to publish my code" argument is complete bollocks.

The code should come first and your advisor should pressure you to publish that first (assuming it actually works).

If someone wants to read your poorly worded explanation of how it works, fine, publish a paper.

Show me the code first.

Re:Differences between the word2vec paper and its implementa... by namibj on Monday June 03, 2019 @08:14PM (#20090453)
I'd much prefer some code with a license that doesn't make looking at it a hazard for writing (A)GPL later, than to have to read the paper many times until I figure out how they meant to combine loads of prior research that is only mentioned to exist on the side, without any info on whether it was used in this or not.

Being able to compile in some way and, possibly after some bug-fixing/compiler-pleasing yielding useful results that approach/relate to those in the paper, greatly increases my confidence in the researcher and his work. Pretty code is nice, but it's weighed separately, not affecting the appreciation for the original research work itself. Only the existence of code with an AGPL or less restricted license affects how I perceive the paper/original research.

Re:Differences between the word2vec paper and its implementa... by hoseja on Tuesday June 04, 2019 @02:37AM (#20092478)
Your paper has to be well documented and clean. The code is often more important than your paper.
Re:Differences between the word2vec paper and its implementa... by ur-whale on Monday June 03, 2019 @09:41PM (#20090957)
You might want to read up on docker.