News for hackers. Stuff that matters.
Re:Differences between the word2vec paper and its implementa... by billconan on Monday June 03, 2019 @06:48PM (#20089777)
For the past week I have been frustrated by an opensource code of a deep learning paper. This type of things are so common in academia. The particular code I looked at has missing documentation, hardcoded local paths, broken dataset download links and broken pretrained model download links. I have to fix bugs before the code can run. I'm very curious how did the author run that code with the bugs.

I call them insincerely opensourced projects.

Re:Differences between the word2vec paper and its implementa... by liuliu on Monday June 03, 2019 @06:55PM (#20089825)
Maintaining a pipeline that works takes effort that can be spent somewhere else. As long as it can be reproduced with a little effort from me, I really don't mind. I would rather they spend time to try a lot of different things, build intuitions, and search better backbone structures than maintaining a CI system so that every time their code updates can train imageNet to < 20% top-1 error rate.

That should be left to TensorFlow model zoo or GluonCV model zoo. I simply look at these research "open-source" as reproducible research.

Re:Differences between the word2vec paper and its implementa... by billconan on Monday June 03, 2019 @07:04PM (#20089894)
I'm not asking for much, but the code should be runnable at least.

This is why I think jupyter notebook and things alike are so important as the future publishing media. Reproducibility is very important.

Re:Differences between the word2vec paper and its implementa... by taeric on Tuesday June 04, 2019 @09:50AM (#20094978)
Jupyter notebooks are laughably not reproducible right now. They may get there, but they encourage bad development habits. Actively.
Re:Differences between the word2vec paper and its implementa... by PaulHoule on Monday June 03, 2019 @06:56PM (#20089828)
Yeah but I wonder what fraction of CS papers test the algorithm that they thought they were testing...
Re:Differences between the word2vec paper and its implementa... by seanmcdirmid on Monday June 03, 2019 @09:33PM (#20090904)
A lot of CS papers aren’t testing algorithms at all.
Re:Differences between the word2vec paper and its implementa... by opportune on Tuesday June 04, 2019 @12:59AM (#20092045)
It is very common in the last few pages of a CS paper to benchmark your results against other algorithms or models
Re:Differences between the word2vec paper and its implementa... by seanmcdirmid on Tuesday June 04, 2019 @08:49AM (#20094428)
And it’s also very common not to. CS papers are pretty diverse in what they are talking about, many aren’t algorithm papers.
Re:Differences between the word2vec paper and its implementa... by colechristensen on Monday June 03, 2019 @06:55PM (#20089822)
It is very common. Scientist code isn't software engineering code (and software engineering code is very often worthy of criticism)

It's just good news when you can find source code at all instead of just being told vague things about something the author did.

Re:Differences between the word2vec paper and its implementa... by itronitron on Tuesday June 04, 2019 @05:52AM (#20093466)
In my opinion, a paper should describe the contribution in sufficient detail that it can be implemented by someone with similar expertise. I believe that would foster greater innovation as people would necessarily have greater familiarity with all of the details, and they would better see why the thing works or doesn't.
Re:Differences between the word2vec paper and its implementa... by raverbashing on Tuesday June 04, 2019 @02:31AM (#20092454)
> Scientist code isn't software engineering code (and software engineering code is very often worthy of criticism)

Absolutely correct, and not in a good way

Though to be fair, people doing research have other interests besides maintainable code (though half or more of the annoyances will probably come and bite them later)

Re:Differences between the word2vec paper and its implementa... by curl_e on Tuesday June 04, 2019 @01:54PM (#20097506)
The good news is people are trying to address this. Starting in the UK there is a movement for Research Software Engineering https://rse.ac.uk/ and happening right now is the German RSE conference https://www.de-rse.org/en/conf2019/
Re:Differences between the word2vec paper and its implementa... by dTal on Tuesday June 04, 2019 @04:05AM (#20092941)
>It's just good news when you can find source code at all instead of just being told vague things about something the author did.

This is extremely common, and I don't see why it should be allowed. I could easily write a paper claiming some kind of interesting result, fake a few "ground truth vs my algorithm" pairs, and talk a bit of mathy-sounding piffle about what it does - and it would be indistinguishable from many papers I have read.

In my opinion - provide runnable source, or what you have isn't a paper, it's a boast.

Re:Differences between the word2vec paper and its implementa... by nickpsecurity on Monday June 03, 2019 @06:58PM (#20089846)
One explained to me here that the problem is they're mainly rewarded by funding agencies for how many papers they publish. That's why quality went down in general. Then, since funding agencies aren't rewarding code, they're either not encouraged to get it right or actively discouraged from doing it to focus on next paper.

This sad situation is both bad for open source and science. Bad for science since inaccurate results that probably aren't even reproducible aren't the knowledge advancements we need. Funding agencies like NSF need to change their policies to address the root cause here. Universities will follow where the incentives lead them.

Re:Differences between the word2vec paper and its implementa... by not2b on Monday June 03, 2019 @07:20PM (#20090003)
Then some other academic can get credit for publishing a rigorous paper describing the actual algorithm, explaining why the original paper is wrong (without speculating on the motive, just the facts), and why the approach taken in the code is in fact better. Bonus points for describing a further improvements, with working code.
Re:Differences between the word2vec paper and its implementa... by YeGoblynQueenne on Monday June 03, 2019 @07:40PM (#20090198)
In machine learning, you don't get credit for publishing rigorous papers. You get credits for publishing papers that show improved performance:

One big challenge the community faces is that if you want to get a paper published in machine learning now it's got to have a table in it, with all these different data sets across the top, and all these different methods along the side, and your method has to look like the best one. If it doesn’t look like that, it’s hard to get published. I don't think that's encouraging people to think about radically new ideas.

Now if you send in a paper that has a radically new idea, there's no chance in hell it will get accepted, because it's going to get some junior reviewer who doesn't understand it. Or it’s going to get a senior reviewer who's trying to review too many papers and doesn't understand it first time round and assumes it must be nonsense. Anything that makes the brain hurt is not going to get accepted. And I think that's really bad.

Geoff Hinton interview:

https://www.wired.com/story/googles-ai-guru-computers-think-...

I guess people look at statistical machine learning and deep learning, see all the formulae and hear all the calculus terminology and think - "oh, wow, that's a really rigorous field! Look at all the formalisms!".

But it's not. It's an extremely, almost exclusively, empirical field. The mathiness and the formulae are just unfortunate attempts to pass off the whole endeavour as something that it's not- some kind of careful science that uncovers deep truths about intelligence and cognition. In truth, it's all just about beating other peoples' systems in very specific benchmarks.

If it wasn't for this culture of pretensions to higher science, machine learning papers would most likely be written with much more clarity than they are now and mistakes like the one described in the above article would be rare.

Re:Differences between the word2vec paper and its implementa... by ur-whale on Monday June 03, 2019 @09:57PM (#20091044)
>It's an extremely, almost exclusively, empirical field.

Fully agree, and it's a necessary disease in young fields like ML (akin to grid search in fact).

But at some point, there will need to be some sort of theoretical foundation brought to bear or advancement will grind down to a halt

And the academic reward mechanism needs to start reflecting that fact.

Re:Differences between the word2vec paper and its implementa... by inimino on Tuesday June 04, 2019 @01:18AM (#20092136)
I've been quite disappointed to see how many papers from "cutting edge" research groups are chasing small improvements in well-known benchmarks by finding new techniques that happen to work, while there is a lot less effort put into finding out why. I guess Geoff Hinton's explanation about what gets published today explains it.
Re:Differences between the word2vec paper and its implementa... by patrick5415 on Tuesday June 04, 2019 @03:49AM (#20092842)
Yet there are approximately 0 papers doing that. Flawed or not not, the original paper still beats the socks off your approach on novelty. And (perceived) novelty is the lifeblood of the acedemic.
Re:Differences between the word2vec paper and its implementa... by kaitai on Tuesday June 04, 2019 @12:06PM (#20096302)
There is a phenomenon in academia I call the "pissing on a lamppost" phenomenon. A well-known researcher mentions his thoughts on (topic) in the penultimate paragraph of a paper on something else. When you talk to them, they say "oh, I don't actually know how to do that, but I think it should work!" So some sucker like me goes & figures out how it actually works, but... BigName has already discussed it so it's not a new result (see citation for penultimate paragraph) so we are not interested in publishing it :)

You can get around it if you show BigName is wrong but if they've been vague enough that's hard.

Re:Differences between the word2vec paper and its implementa... by throwaway287391 on Monday June 03, 2019 @07:26PM (#20090058)
This sort of thing is aggravating to read. Frankly it comes off as really entitled. As researchers, the expectation is now that we not only have to do the research and write a paper like the good old days, but we have to release the code too. Okay, fine. But now that's not enough either -- the code has to be well-documented and clean. Ugh, alright, fine -- it's going to take me a few extra weeks of not doing research, but I'll clean up all the code, rerun experiments to make sure it all still works like before, and add a bunch of documentation. But no, still not good enough -- it has to run at the press of a button in your particular programming environment. If we don't know how to write a script (or couldn't be bothered to spend the time writing one) to check that the data is on disk and, if not, crawl a website to download some huge dataset in one click, test our code on your OS, your CPU/GPU/TPU/..., etc., we were being "insincere" with our open-sourcing efforts.
Re:Differences between the word2vec paper and its implementa... by KirinDave on Monday June 03, 2019 @08:58PM (#20090717)
Pardon me, maybe I just misunderstood the whole idea of research but what good is it if it's not reproducible?

I can understand it may be part of a meaningful personal journey for you, and I appreciate that. But if no one else can validate your research they're correct to discredit it and you.

So what is the optimal outcome here? Should we hold you to a standard of reproducibility even if it is as minimal as, "actually describe your algorithms correctly and don't misrepresent a piece of code and a paper?" Or should everyone just decide you can find your own research funding if it's not going to help anyone?

Re:Differences between the word2vec paper and its implementa... by archgoon on Monday June 03, 2019 @09:37PM (#20090931)
The original idea behind "reproducible" is that the ideas conveyed in the paper should be enough to reproduce the results. Physicists and biologists are not expected to drive over to your lab to figure out what's wrong with your setup.

Now, that said, reproducibility is terrible in many fields. CS has an opportunity to act as a trailblazer here, but it should be noted that this would be holding themselves to a higher standard than their peers in other fields. As a result, there's going to be a learning process for everyone as they figure out how to make this all work. :)

Re:Differences between the word2vec paper and its implementa... by throwawayjava on Monday June 03, 2019 @10:20PM (#20091175)
Some pretty good computer science got done before devops was gifted to the world.

And some pretty good science got done before computer scientists were gifted to the world.

I'm genuinely skeptical that modern software engineering practices are a good way of thinking about reproduction in science. Even in computer science. There's a lot that scientists can learn from software engineering (and in fact I've helped run workshops in the past on exactly this topic), but science is not engineering.

Re:Differences between the word2vec paper and its implementa... by KirinDave on Wednesday June 05, 2019 @10:51AM (#20104983)
> Some pretty good computer science got done before devops was gifted to the world.

I'm happy to talk about this if you want. One of the most important aspects of this work was that people like Dijkstra started using notions that approached what real computers could read while remaining human-readable. This is some measure of classical "reproducibility". And work like McCarthy's was revolutionary in part because it was a definition of reproducibility as a result!

I can give examples of shockingly good papers that are struggling to see the light of day in their industry because they're written in ways that make them hard not only to understand, but to reproduce.

So don't presume to lecture me about this. Part of the reason the word2vec paper stands out is precisely because this is such a deviation from the norm to have a paper misrepresent its most fundamental component: the algorithm.

Re:Differences between the word2vec paper and its implementa... by KirinDave on Wednesday June 05, 2019 @10:35AM (#20104798)
Okay, by the standard you laid out: the paper that is the subject of this paper failed.

But similarly, if you say, "Here are our statistical models, they're in a notation for a private, custom MCMC system you can't use" that's probably failing the standard even if the work is good.

Re:Differences between the word2vec paper and its implementa... by seanmcdirmid on Monday June 03, 2019 @09:28PM (#20090884)
If I have to go dig up someone’s code to reproduce their result because I don’t believe/understand the idea in its published form, they’ve already lost me.

Word2vec is a great example of a piece of research that conveys a great idea with plausible results, where in fact, the numbers are less useful than the story behind them. Of course, google is a big corp that can just put out useful interesting stuff without worrying about how to play the science game.

Re:Differences between the word2vec paper and its implementa... by p1esk on Monday June 03, 2019 @09:23PM (#20090857)
Pardon me, maybe I just misunderstood the whole idea of "reproducible" but it is only not reproducible if you tried to implement the ideas described in the paper yourself, and I mean really tried, and contacted the authors for help, and still failed to achieve the claimed results.
Re:Differences between the word2vec paper and its implementa... by throwawayjava on Monday June 03, 2019 @09:47PM (#20090995)
> if you tried to implement the ideas described in the paper yourself, and I mean really tried, and contacted the authors for help, and still failed to achieve the claimed results.

This is the actually important definition of "reproducible".

"has an install.sh" is really nice, but it's far more important that an informed reader can recreate the artifact for themselves from the written description.

The "must be able to apt install or it's not real science" is a particularly dangerous path to go down. It's how you end up with e.g. shit loads of well-engineered LISP or FORTRAN code with no actual scientific insights or knowledge transfer. Which has actually happened in the past. A lot.

Re:Differences between the word2vec paper and its implementa... by telchar on Tuesday June 04, 2019 @11:06PM (#20101153)
I don't think people in this thread are insisting on a downloadable package. Just that the algorithm published in the write-up accurately describe the work that was actually done to achieve the stated results. If even that threshold can't be met, IMO the authors are polluting the literature with bad work.
Re:Differences between the word2vec paper and its implementa... by derefr on Monday June 03, 2019 @11:41PM (#20091648)
> and contacted the authors for help

Isn’t the whole point of a paper, to obviate needing to talk to the authors? Like, so that science is a ratchet that doesn’t slip backward the moment the authors die?

Re:Differences between the word2vec paper and its implementa... by visarga on Tuesday June 04, 2019 @01:59AM (#20092306)
Related to w2v, the proof is in the vectors. They work. And, by the way, w2v has had a whole family of variants which were built on top of the original implementation (such as Doc2Vec, Doc2VecC and others).

Related to most ML code, it's not written in C but in Python frameworks that implement whole layers as building blocks. So it's not as hard to read/change. You can take something complex like the transformer and understand it in a few minutes. The most puzzling part with framework code is figuring out the shape of tensors and, sometimes, what each dimension means.

Regarding reproducibility - it's hard to achieve on account of parallelisation. You see, neural nets use floats, and floats are not real numbers. Half the float numbers lie between [-1, 1] and the rest outside. If you do 1e-10+1e10 you get 1e10. So if you add 1000 times 1e-10 to 1e+10 you still get 1e+10, but if you add them up together first, the result is different. So float summation is not commutative. Depending on race conditions the order in which summation occurs could change and the result change as well, even if you use the same random seeds. And if we give up parallelism then we can't run the experiments any more.

Re:Differences between the word2vec paper and its implementa... by Dylan16807 on Tuesday June 04, 2019 @07:30AM (#20093921)
For almost all ML training, it's straightforward to break it into a fixed number of chunks without race conditions.

That's if float rounding issues are even a big problem in the first place. If your results are within .1% over a few runs it's reproducible enough for most purposes.

Re:Differences between the word2vec paper and its implementa... by throwawayjava on Monday June 03, 2019 @09:44PM (#20090977)
> what good is it if it's not reproducible?

I am one of those people who actually did the extra weeks/months to properly test/review/document/release my code and data sets (you can apt-get install my "research artifacts").

In retrospect, it was a poor use of my time and a poor use of my sponsoring institution's time. "apt-get install" is NOT what we mean by "reproducible" in science.

High-quality or easy to install code is not necessary for a result to be reproducible. True reproduction would mean coding the algorithm from scratch by following the written description, and that's what it literally means in most other fields of science.

You can't download & install a Large Hadron Collider in an afternoon. Does that mean the LHC experiments are "not reproducible"? Of course not.

But that's not even the important point. The really important point is that, in most cases, high-quality code is not even sufficient for a result to be reproducible! See: the article we're discussing.

IMO, the "blindly rerun the code" definition of reproduction is actually a HUGE barrier to creating a true culture of reproducability in computer science. It results in super lazy reviewing where "public source code that's easy to install and puts the correct-sounding shit into STDOUT" becomes a stand-in for "paper actually describes a novel idea in enough detail that it can be truly reproduced".

> So what is the optimal outcome here?

An optimal allocation of scientists' time and effort.

As a scientist who has actually done that leg work, I don't think packaging code so that it runs with a single click is the best use of public money in science in 99.9999% of cases. That time is much better spent on writing and other dissemination explaining the ideas that make the code work (in some cases well-documented source code is the best description but in other cases prose is much more effective and illuminating). Or on coming up with new ideas that are even better than the old ones.

Which I guess is just another way of saying that scientists should spend their time on science, not engineering.

P.S. When shitting on "scientists" for not being good enough software engineers, please remember who's going to be doing the actual work you're demanding. It's mostly phd students who make $30K/yr. And they have to do this work in their free time because their 60 hr/wk day job is fully allocated to doing the actual science. I.e., treat scientists who maintain their code as you would treat FOSS contributors who are making 5x-10x+ less than you while working longer hours. Because maintaining high-quality code is something they are almost certainly doing in their free time.

Re:Differences between the word2vec paper and its implementa... by KirinDave on Wednesday June 05, 2019 @10:46AM (#20104929)
> High-quality or easy to install code is not necessary for a result to be reproducible. True reproduction would mean coding the algorithm from scratch by following the written description, and that's what it literally means in most other fields of science.

Which failed here. However, you're going to have a difficult time convincing many people that the attributes you described would be bad properties, just that they might not justify their expense.

> You can't download & install a Large Hadron Collider in an afternoon. Does that mean the LHC experiments are "not reproducible"? Of course not.

By the same token though, the LHC repeats experiments and solicits feedback on how to improve their methods, which they go to great lengths to publish and simulate, because they're aware of this problem.

> IMO, the "blindly rerun the code" definition of reproduction is actually a HUGE barrier to creating a true culture of reproducability in computer science. It results in super lazy reviewing where "public source code that's easy to install and puts the correct-sounding shit into STDOUT" becomes a stand-in for "paper actually describes a novel idea in enough detail that it can be truly reproduced".

Ah, yes. Yes. "If this code is TOO reproducible then people might reproduce it, and handwave handwave the quality of papers would decline.

That's certainly NOT the case in pure CS papers, which have only improved since the days when folks felt that "Lenses, Bananas and Barbed Wire" was how folks should go about writing papers.

Now, physics might be different. But there is surely a middle ground between, "I've shipped you a LHC just plug in in lol" and "This paper doesn't even remotely describe how we achived the results."

If you believe that wasting the time of scientists is bad, then surely you're for clear papers with accurate descriptions of the methods so that those who go and reproduce your work are not sent on wild goose chases?

> As a scientist who has actually done that leg work, I don't think packaging code so that it runs with a single click is the best use of public money in science in 99.9999% of cases.

No, we got that part. But surely someone does and maybe you can design your work to leverage that rather than reproducing and discarding scaffolding. My big concern here is that a lot of scientists (like you claim to be) are underqualified and unpracticed at software, and thus are surely seeing at least some aspect of their work distorted by software and hardware issues.

> Which I guess is just another way of saying that scientists should spend their time on science, not engineering.

Scientists are not going to be able to escape engineering. No one else is going to build what they need besides them.

> please remember who's going to be doing the actual work you're demanding. It's mostly phd students who make $30K/yr. And they have to do this work in their free time because their 60 hr/wk day job is fully allocated to doing the actual science.

Yeah, I'm aware. I suspect their lot would be better if your attitude wasn't that their work is disposable and unimportant.

Re:Differences between the word2vec paper and its implementa... by patrick5415 on Tuesday June 04, 2019 @03:21AM (#20092669)
Reproducible research does not mean you don’t have to do any work. The original author isn’t going to come to your lab and clean your pipettes nor should they. Grow up.
Re:Differences between the word2vec paper and its implementa... by cfcf14 on Tuesday June 04, 2019 @05:36AM (#20093389)
The research published in this case (the original word2vec paper) is quite literally not what it claims to be. It is factually incorrect. Is that really the responsibility of the individual attempting to reproduce the work to identify and correct?

How wrong should the original work be before it ceases to be a case of 'clean your own pipettes'?

Re:Differences between the word2vec paper and its implementa... by patrick5415 on Tuesday June 04, 2019 @12:49PM (#20096809)
I’d say at the same point that others in the field would generally agree it’s fradulent. I’m not familiar enough with the field of word2vec to say if this is the case or not. But regardless, asking that the implementation be representative of the paper is a lot different than demanding docker containers and install scripts.
Re:Differences between the word2vec paper and its implementa... by KirinDave on Wednesday June 05, 2019 @10:48AM (#20104948)
Patrick,

Could you PLEASE consider reading the article we're all discussing before you roll into the comments section of an article about it with your strong-but-loosely-held opinions? You're arguing against a point that almost no one is putting forward (that software should be one click).

Re:Differences between the word2vec paper and its implementa... by liquidify on Monday June 03, 2019 @07:42PM (#20090221)
If there are bugs in your code that would have prevented you from being able to do what you claim to do, then to the readers of your paper, it seems more likely that you might not have performed the experimentation that you claim to.
Re:Differences between the word2vec paper and its implementa... by throwaway287391 on Monday June 03, 2019 @07:50PM (#20090273)
What? This has nothing to do with bugs in the code that would've affected the original author's ability to run the original experiments described in the paper. The user I replied to is talking about things like hard-coded data paths and wanting to be able to run the code at the press of a button on any platform. The expectation for this level of cross-platform compatibility is difficult to achieve for engineering firms with millions to invest in making it work properly. He's mad that individual machine learning researchers don't always do a great job of making it happen for their cutting-edge research which was a difficult enough engineering task to get running on their machine.
Re:Differences between the word2vec paper and its implementa... by gcommer on Monday June 03, 2019 @07:59PM (#20090344)
Reread the original post:

> I have to fix bugs before the code can run. I'm very curious how did the author run that code with the bugs.

Re:Differences between the word2vec paper and its implementa... by throwaway287391 on Monday June 03, 2019 @08:17PM (#20090473)
You're right, I misread that part.
[dead]
Re:Differences between the word2vec paper and its implementa... by dang on Tuesday June 04, 2019 @12:15AM (#20091840)
Personal attacks will get you banned here. Would you please review the site guidelines and not post like this again?

https://news.ycombinator.com/newsguidelines.html

Re:Differences between the word2vec paper and its implementa... by jonas21 on Monday June 03, 2019 @11:00PM (#20091368)
FWIW, I downvoted because because you somehow managed to break about half of the HN comment guidelines [1] in a single comment, in particular:

* Be kind. Don't be snarky. Comments should get more thoughtful and substantive, not less, as a topic gets more divisive.

* When disagreeing, please reply to the argument instead of calling names. "That is idiotic; 1 + 1 is 2, not 3" can be shortened to "1 + 1 is 2, not 3."

* Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith.

* Please don't make insinuations about astroturfing. It degrades discussion and is usually mistaken. If you're worried, email us and we'll look at the data.

* Please don't comment about the voting on comments. It never does any good, and it makes boring reading.

[1] https://news.ycombinator.com/newsguidelines.html

Re:Differences between the word2vec paper and its implementa... by throwaway287391 on Monday June 03, 2019 @09:49PM (#20091005)
I don't have much to say to most of this, but I'll point out that I didn't and couldn't have downvoted you as none of my throwaways have >=500 karma or whatever the threshold is.
[dead]
Re:Differences between the word2vec paper and its implementa... by throwawayjava on Monday June 03, 2019 @10:46PM (#20091298)
Man, I hate when I get dragged into these sorts of intense emotional downs by silly internet conversations with strangers.

FWIW I undowned your posts; internet karma is about communication/moderation, and sometimes has the opposite of the intended effect. I've been there.

Anyways, find some time to pet a dog/cat or fly a kite today.

Re:Differences between the word2vec paper and its implementa... by p1esk on Monday June 03, 2019 @09:38PM (#20090942)
Typically when I post code for my papers, I try to clean it up, because it's an ugly mess that is hard to understand (even myself sometimes). This clean up process might introduce bugs. I'm not going to thoroughly test everything, I have next paper deadline coming up and my adviser could not care less about my code. But, if you're interested in my research and having trouble reproducing my results, email me and I'll help you.
Re:Differences between the word2vec paper and its implementa... by throwawayjava on Monday June 03, 2019 @10:50PM (#20091318)
...and university/company sometimes makes you wipe out the git commit log, so you can't always just use "HEAD at time of submission".

Pro tip: at paper submission time, md5sum your code and also git tag it in your private repo. When you release the code, if you have to / want to release with a clean history, make submission-time your initial commit and then make the current state of the repo your second commit. I've never encountered an institution that won't allow that level of history in the code release, even places that are pretty hard core about wiping pre-release history.

Re:Differences between the word2vec paper and its implementa... by caenorst on Monday June 03, 2019 @11:01PM (#20091371)
And don't you think it would have been more efficient to write it down clean at first with good comments and code practice ? Those good practices are here for a reason and not just for annoying you for the sack of "beauty". You would save more time for yourself when you wanna expand the paper further or just reuse some part of it, have better reputation for your work, and gain some citation because people will have the tendency to use your code as a starting point.
Re:Differences between the word2vec paper and its implementa... by inimino on Monday June 03, 2019 @11:21PM (#20091519)
"Writing it down clean at first" implies that you know what is going to work when you start. If that's the case, it's not research.
Re:Differences between the word2vec paper and its implementa... by patrick5415 on Tuesday June 04, 2019 @03:39AM (#20092779)
No, it would not be more efficient. I’ve tried this route and it hands down consumes way more time. Until it’s clear that something will get re-used, it’s most efficient to leave it dirty.

The software types love to sht on researchers for their poor quality code. Yet a few threads over, there’s always a discussion about how more tests don’t make it onto the kanban board (or whatever you guys are using these days) when crunch time approaches. It’s not just us.

Re:Differences between the word2vec paper and its implementa... by randcraw on Monday June 03, 2019 @07:57PM (#20090330)
Agreed. If your results are not reproducible by others, the publication should issue a retraction and withdraw your work. Code that works is essential.
Re:Differences between the word2vec paper and its implementa... by dodobirdlord on Monday June 03, 2019 @11:59PM (#20091755)
Nonsense! The "reproducible" in "reproducible result" doesn't mean anything except that what is described in the paper can be reproduced. Code that works is nice and all, but totally irrelevant to the question of reproducibility. Would you claim that that the Higgs paper must be retracted because CERN won't ship you the LHC? They've got ATLAS and CMS to demonstrate reproducibility, and they're wildly different!
Re:Differences between the word2vec paper and its implementa... by Volt on Monday June 03, 2019 @10:54PM (#20091344)
I hope you're a scientist.
Re:Differences between the word2vec paper and its implementa... by Veedrac on Monday June 03, 2019 @07:48PM (#20090264)
If it was just one thing or the other, perhaps he wouldn't have been so critical of it. People are going to understand if you require a few paths to be changed or some files to be downloaded manually into specific relative locations. The ask is only that somebody should be able to use the code without doing their own novel research, and be able to verify it's doing what it is claimed to be doing. If this means your download links need to point to the right location, the code has to run in a location other than /home/hellokitty/ml, and that it needs to meet standards of readability—well, good, that's what the scientific process is about.
Re:Differences between the word2vec paper and its implementa... by caenorst on Monday June 03, 2019 @07:58PM (#20090338)
I have spent enough time to debunk papers to know that a lot of researchers are shamelessly cheating or make mistakes (as EVERYBODY does). Reviewing your code should be part of the peer review process.

Sorry but I don't trust your research just based on yours words and cherry picked images.

Also research is incremental, so producing a proper code on which other can work on top of should be part of the CONTRIBUTION.

Re:Differences between the word2vec paper and its implementa... by ur-whale on Monday June 03, 2019 @09:52PM (#20091020)
I fully agree, and would actually go a lot further.

A very large number of research papers I've read in my life were so poorly written as to being near impossible to understand.

The time it took to infer the actual meaning of certain parts of certain papers was vastly larger than just reading the code would have been.

It has become very clear to me that being a strong CS researcher does not necessarily imply mastery of language or even the very basic ability to explain one's ideas. As a matter of fact, with very few exception (eg Feynmann), the two skills seem to be at odds with one another.

On the other hand, code can be read and understood much more readily, especially when it can be experimented with, if only to insert printfs in it to understand what it does.

The culture of scientific publication in CS must change. The "I'm pressured by my advisor and therefore have no time to publish my code" argument is complete bollocks.

The code should come first and your advisor should pressure you to publish that first (assuming it actually works).

If someone wants to read your poorly worded explanation of how it works, fine, publish a paper.

Show me the code first.

Re:Differences between the word2vec paper and its implementa... by namibj on Monday June 03, 2019 @08:14PM (#20090453)
I'd much prefer some code with a license that doesn't make looking at it a hazard for writing (A)GPL later, than to have to read the paper many times until I figure out how they meant to combine loads of prior research that is only mentioned to exist on the side, without any info on whether it was used in this or not.

Being able to compile in some way and, possibly after some bug-fixing/compiler-pleasing yielding useful results that approach/relate to those in the paper, greatly increases my confidence in the researcher and his work. Pretty code is nice, but it's weighed separately, not affecting the appreciation for the original research work itself. Only the existence of code with an AGPL or less restricted license affects how I perceive the paper/original research.

Re:Differences between the word2vec paper and its implementa... by hoseja on Tuesday June 04, 2019 @02:37AM (#20092478)
Your paper has to be well documented and clean. The code is often more important than your paper.
Re:Differences between the word2vec paper and its implementa... by ur-whale on Monday June 03, 2019 @09:41PM (#20090957)
You might want to read up on docker.
Re:Differences between the word2vec paper and its implementa... by krageon on Tuesday June 04, 2019 @04:36AM (#20093105)
I had the great displeasure to work on multiple state-of-the-art algorithms related to sticking things in buckets with varying degrees of optimisation and/or supervision. There are a ton of reasonably recent papers for that, some of them have source code (most of them have nigh-impenetrable theory on why their algorithm is the best) and every case I tested was emphatically much worse than the technologies we've been using for 10+ years in any practical sense.

That problem is compounded with the issues you describe, which I'd categorise as probably the best case. I did not find even a single case of a properly opensourced repository that just compiled somewhere that wasn't the researcher's laptop.

Re:Differences between the word2vec paper and its implementa... by nullc on Monday June 03, 2019 @11:36PM (#20091612)
There appears to be something of a software engineering crisis in academia.

Imagine that you are a graduate student or a newly minted PHD who happens to be pretty good at practical software engineering: You can leave academia for FAANG and a solid six figure income, or you can stay struggling, poorly paid, unable to get tenure in academia. In some groups software engineers are treated as glorified typists, too valuable enabling other people's work to be a principle researcher themselves. I've heard from a number of people that they avoid hiding their programming skills to avoid that trap.

This has many consequences beyond the obvious direct ones like papers with unusable implementations.

Academic work is often valuable even when it isn't practically useful but in many cases where academics are /trying/ to do something practical they fail because their community lacks the engineering experience-- I've seen a fair number of papers presenting optimizations as useful engineering when in reality their approach only makes sense because of inefficient non-programmer tools make a weird set of primitives fast (e.g. using a matrix multiply in matlab where in C you'd simply write a loop). As a practitioner this is frustrating because it sometimes requires re-implementing the approach to realize that it was only 'fast' compared to a pants-on-head-foolish approach.

It also can create essentially fake results.

Many times the lack of software engineering is compensated for by mocking components out in ways that would give faithful results if the researcher understood everything, but the whole point of expirementation is that the researcher doesn't! This mocking is seldom disclosed in papers. For example, I've encountered multiple papers that claimed to implement some enhancement to Bitcoin and tested it in a test bed but in reality they just added sleep()s with appropriate times, guessed out of specious reasoning like "our function X should be 10x slower than a signature", and hardcoded constants (computed in mathmatica or sage) for their messages. ... not realizing that sleep() isn't the best analog for cpu busywork in a multithreaded program!

Another kind of fake result I've encountered which is even less directly a result of the software engineering shortage is in signal processing literature. While working on audio/video compression I found it common for algorithms to be presented without various constants and after reimplementing and asking the authors for their constants I found that they'd been cherrypicked for the ten images used in the paper, and that the whole approach doesn't actually work. This is a kind of ineptitude (or outright dishonesty) that would be much less common in a world where reviewers received a working and usable implementation in source form-- but that can't be expected in a world where qualified software engineering is not readily available to researchers.

I don't have any proposed solutions but I think it's important to acknowledge that it is a common and serious limitation to the usefulness and accuracy of contemporary research.

Re:Differences between the word2vec paper and its implementa... by amelius on Tuesday June 04, 2019 @04:34AM (#20093088)
I think there should be more software engineering in academia for different reasons too.

In the past academia and government agencies developed the internet and its communication protocols; this was (imho) a much better internet than the corporation-dominated internet we have today.

I'd like to see a resurge of this type of internet. Let us use well-researched open protocols again; and let us use applications which are really built for people, not advertisers. Let corporations build the hardware, but let us keep our data far away from them.

And besides the computer science / software engineering branches, I'd like to see other fields (like industrial design, and for instance even sociology) to join the development of a better digital future. The kind of future where everybody profits, not just shareholders of big companies.

Re:Differences between the word2vec paper and its implementa... by izacus on Tuesday June 04, 2019 @07:43AM (#20094001)
> Another kind of fake result I've encountered which is even less directly a result of the software engineering shortage is in signal processing literature. While working on audio/video compression I found it common for algorithms to be presented without various constants and after reimplementing and asking the authors for their constants I found that they'd been cherrypicked for the ten images used in the paper, and that the whole approach doesn't actually work. This is a kind of ineptitude (or outright dishonesty) that would be much less common in a world where reviewers received a working and usable implementation in source form-- but that can't be expected in a world where qualified software engineering is not readily available to researchers.

This happened so many times when I was working on NLP/CV algorithms - I've read and implemented many algorithms from papers just to find out that they only produce the amazing improvements on cherrypicked dataset. On pretty much all other practical data the algorithms performed worse and in many cases even crashed!

Re:Differences between the word2vec paper and its implementa... by kaitai on Tuesday June 04, 2019 @12:00PM (#20096235)
I just wish I could upvote your comment three more times.

To compound the problem, the fact that you're competing against many of your peers for the few jobs there are, and jumping positions every 1-3 years after grad school, means that best practices are not disseminated by working with a stable set of coworkers. This is similar to your 4th paragraph point.

As a mentor of grad students on projects that these days are increasingly using machine learning, I am trying to figure out how to add this stuff to my teaching load, essentially, because the pain of my students using nested for loops when they could just use a single vectorized expression is real... but I myself have no formal CS training......

Re:Differences between the word2vec paper and its implementa... by GuyPostington on Monday June 03, 2019 @06:52PM (#20089800)
Have you been able to reach out to the author and get clarification?
Re:Differences between the word2vec paper and its implementa... by billconan on Monday June 03, 2019 @06:53PM (#20089816)
I tried to file bugs on github, only to find out that people (sometimes 2 years ahead me) was blocked by the same issue and there are no answers.

They didn't even bother to merge my pull requests.

Re:Differences between the word2vec paper and its implementa... by scott_s on Tuesday June 04, 2019 @03:44PM (#20098538)
There's probably a disconnect in expectations. The researchers are probably publishing the code more as a document, to say, hey, this is how we did this. It sounds like you have expectations that the code is part of an ongoing project. But ongoing projects require maintenance, and the researchers have likely moved on to new things.
Re:Differences between the word2vec paper and its implementa... by IfOnlyYouKnew on Tuesday June 04, 2019 @09:47AM (#20094952)