News for hackers. Stuff that matters.
Re:Should All Research Papers Be Free? by mmmBacon on Sunday March 13, 2016 @08:32PM (#11280199)
I also think code used to produce results should be published as well.
Re:Should All Research Papers Be Free? by Balgair on Sunday March 13, 2016 @11:47PM (#11280933)
Have you worked with biologists? If they could figure out how to use git, let alone github, I would be astounded. No forgetting, these are incredibly smart people, they just are bench-top scientists, not laptop ones. If you want their code, it will be, at best, a copy-paste of badly hardcoded Matlab, years in the making, with for-loops for the sake of for-loops alone, and without ANY comments or documentation whatsoever. Honestly, it would take less time for you to write it yourself and compare, rather than to try to peek under the hood of a spaghetti mess.

Honest to god and on the grave of my mother this happened to a friend. He was going over some old Fortran 88 code, filled with GOTO statements. The code, at best, was a rat's nest. Only a deep and long fight could get it into your brain. At about 3 am after a long day in the lab, she finally gets to a line in the code that says GOTO LINE 12345 with the comment 'HAHA MADE YOU LOOK' . Her boss bought her a new computer after she threw that one off the roof.

That experience is universal with Biologist code.

Re:Should All Research Papers Be Free? by AnthonyMouse on Monday March 14, 2016 @12:16AM (#11281030)
> Honestly, it would take less time for you to write it yourself and compare, rather than to try to peek under the hood of a spaghetti mess.

That's not actually the point. The point is reproducibility. If you write your own code and get a different result, why? If the reason is the code then having the other code, no matter how terrible it is, allows you to figure out what the difference is. And then somebody is wrong and you know why and can publish that result.

Re:Should All Research Papers Be Free? by timr on Monday March 14, 2016 @07:12AM (#11282147)
Exactly. IMO, this is a big problem in some areas of computational biology right now. Code to a famous project gets released, and instead of independently validating the results, other researchers (rationally) choose to just build on top of the famous project. It's faster, easier, and the "collaboration" leads to social prestige for everyone involved.

The end result is that empires are built, reproducibility is (paradoxically) harmed, and we're someday going to end up finding out that some big, high-profile projects were built on pillars of sand.

Re:Should All Research Papers Be Free? by lsc on Monday March 14, 2016 @01:21AM (#11281216)
My experience is that it takes a certain kind of twisted intelligence to write code that is below a certain level of quality, but still works. Someone like me who is a mediocre programmer? if my code gets too terrible, it simply stops working. I'm forced to use some base level of software engineering if I want to write a large program that runs, just because there's a limit on how many random side effects I can keep track of.

From what I've seen when supporting EEs? People who are that much smarter than I am don't have this limitation. They can write thousands of lines of spaghetti perl and bash and as long as nobody touches the damn thing, it works fine. God help you, though, if you make a change.

But... there is an established job role for this; I mean, you don't want to make your EEs sysadmin their own LSF cluster, either.

Re:Should All Research Papers Be Free? by collyw on Monday March 14, 2016 @07:41AM (#11282242)
Nope, its always below average programmers who don't seem especially smart, and who don't want to learn who produce the worst code in my opinion.
Re:Should All Research Papers Be Free? by GFK_of_xmaspast on Monday March 14, 2016 @12:33PM (#11283780)
There's a difference between 'being smart' and 'being able to program well', and that difference is precisely why academic code has its reputation.
Re:Should All Research Papers Be Free? by lsc on Monday March 14, 2016 @12:50PM (#11283868)
My observation is that to make a program run, a less intelligent person needs to organize that program better. But, intelligence is a controversial subject in and of itself, so I don't expect to see agreement.
Re:Should All Research Papers Be Free? by drjesusphd on Monday March 14, 2016 @05:52AM (#11281860)
> God help you, though, if you make a change.

Or need to add functionality. Or, just wait a year.

Re:Should All Research Papers Be Free? by cheald on Monday March 14, 2016 @12:32AM (#11281085)
Sounds to me like there's an opportunity to propel biology forward by teaching some biologists how to manage their code in a way that won't cost large amounts of productivity.
Re:Should All Research Papers Be Free? by sitkack on Monday March 14, 2016 @07:46AM (#11282262)
This is exactly the goal of http://software-carpentry.org/workshops/
Re:Should All Research Papers Be Free? by Bluestrike2 on Monday March 14, 2016 @01:46AM (#11281273)
Ha, it could be worse. I have a friend who's getting his PhD in microbiology and he and his team were running into some trouble with these massive multi-gigabyte CSV files from NIH (gene sequencing data, if I recall). Turns out they were trying to open the little buggers in Excel and wanted to stay in Excel since it was familiar. At some point, I wound up just suggesting they use split on the stupid (but very important) things and people acted like I just blew their minds.

I think sometimes you just get so caught up in your field of expertise that you miss some of the tools that could drastically help you entirely. You have to wonder how much of an efficiency drain that adds up to over the course of even just individual research projects. And that's before you get to bad code.

Re:Should All Research Papers Be Free? by JupiterMoon on Monday March 14, 2016 @01:40PM (#11284193)
What is Fortran 88? I was under the impression it jumped from Fortran 77 to Fortran 90?
Re:Should All Research Papers Be Free? by lacker on Sunday March 13, 2016 @09:23PM (#11280426)
That would also have the benefit of making sure the code actually works. From my time in CS grad school I'm pretty sure there are many published papers where the author never had code that achieved the results they described.
Re:Should All Research Papers Be Free? by taneq on Sunday March 13, 2016 @10:18PM (#11280661)
Or code that worked on the specific test case described in the paper but was flaky as hell on anything else.
Re:Should All Research Papers Be Free? by albinofrenchy on Monday March 14, 2016 @09:49AM (#11282848)
The most disillusioning thing for me as an undergrad was to try to replicate results from a paper on a relatively complicated problem, and not achieving anywhere near the papers level of results.

I got access to the source code, and the super complicated algorithm added almost nothing to the results, the glossed over / hand waved past data normalization worked so well that there wasn't a need for any further classification. This paper was pretty well received and cited, despite it basically not work.

Re:Should All Research Papers Be Free? by andreasvc on Monday March 14, 2016 @06:54PM (#11286161)
I am afraid that what you describe is a pretty fundamental limitation of peer review. Currently it is not expected of reviewers to try to replicate experiments, but only to check if the methodology of the paper seems sound and the relevant previous work is cited. Should this be expected of reviewers? Should the authors provide all code and data so that the results can be replicated at the push of a button? That seems a pretty tall order. Seems like we'll be stuck with a modus operandi of lots of papers being published, peer review only catching obvious errors, and replication being very rare.
Re:Should All Research Papers Be Free? by albinofrenchy on Monday March 14, 2016 @08:36PM (#11286672)
I get that, and I don't even think that that should be the reviewers job. But for CS in particular, I do wish more people published source code and build procedures.

I think a central hub for public comments, with source, and available PDF would be incredibly useful for CS as a field. For most papers I had access to, to get the source I'd have to go through at least one person. So I tried for quite a while to replicate the result before even trying to get access. If it was available, on say github, I'd have grabbed the source code as a resource to understand the paper, and with something like publicly available commenting, I think it'd have been a non-issue.

The problem is, I can see how such a system would be incredibly constructive to the field and the community, but it could be a liability to the authors, and would make publishers meaningless, so I doubt it happens anytime soon.

Re:Should All Research Papers Be Free? by arcanus on Sunday March 13, 2016 @10:23PM (#11280680)
At least in the engineering fields, many of the papers require at least a discussion of the verification and validation of the codebase. So it is improving, if admittedly far too slowly.
Re:Should All Research Papers Be Free? by drjesusphd on Monday March 14, 2016 @05:57AM (#11281874)
The problem is it's very difficult to fit a full and proper validation in the appendix of a paper. What's really needed is to standardize the inclusion of "additional material" with the paper, including source code, test cases, and results.
Re:Should All Research Papers Be Free? by studentrob on Sunday March 13, 2016 @11:35PM (#11280892)
Wow, I never considered that this might fall under the same umbrella. It seems very reasonable because it isn't very hard to publish that now. Even if the shared code wouldn't run without a ton of requirements and very specific setup, it's still something. I know we all hate to release code that doesn't compile or run outside our own setup because we then feel we will be responsible for helping someone else getting it to run on their machines to which we have no physical access. Not to mention it is always a mess figuring that stuff out. But other research fields do not feel compelled to hold the hands of their researchers to replicate experiments. Presumably, whoever's code is easiest to run will more likely serve as the basis for other future research, thereby enhancing career growth of the original publisher. I'm not a researcher by the way, and this just dawned on me, so if this is obvious to all of you, then I apologize for the block of text.
Re:Should All Research Papers Be Free? by SalmoShalazar on Sunday March 13, 2016 @09:18PM (#11280405)
I agree with this in theory... but do you really want my garbage, ad hoc, messy code that I used to pipe output from one program to another?
Re:Should All Research Papers Be Free? by mrcsparker on Sunday March 13, 2016 @10:07PM (#11280614)
Yes, please.

To be fair, a lot of people do publish their working code with the paper. It helps people like me understand how to do the work.

Bad code is fine, good code is better, great code is great. Stick to fine and I can do my job and you can write papers knowing that your work will be used by many.

Re:Should All Research Papers Be Free? by skybrian on Sunday March 13, 2016 @09:38PM (#11280507)
I'm not actually recommending this license, but it seems relevant to mention:

The CRAPL: An academic-strength open source license http://matt.might.net/articles/crapl/

Re:Should All Research Papers Be Free? by marvy on Sunday March 13, 2016 @10:13PM (#11280634)
"the Community Research and Academic Programming License"
Re:Should All Research Papers Be Free? by cube00 on Sunday March 13, 2016 @09:28PM (#11280458)
Definitely yes, then I can see if somewhere in that garbage, ad hoc, messy code why I can't replicate the result myself.
Re:Should All Research Papers Be Free? by zmmmmm on Sunday March 13, 2016 @10:10PM (#11280624)
If it's that garbage, how do you know your results are correct? It should be at least non-garbage enough that someone else could theoretically check it, or you're not doing science at all.
Re:Should All Research Papers Be Free? by jjoonathan on Sunday March 13, 2016 @10:19PM (#11280665)
You don't. But there are 5x-100x more bugs lurking in the methodology so triage is the name of the game.

A much bigger problem is that grantsmanship strongly incentivizes against verifying your methodology and being diligent in your construction of null hypotheses.

Re:Should All Research Papers Be Free? by euske on Sunday March 13, 2016 @11:18PM (#11280848)
This is like saying "I don't want to file a tax because my handwriting is bad".

Well, if you're embarrassed by your terrible handwriting, maybe it's a time to put some effort to improve it.

Re:Should All Research Papers Be Free? by dinkumthinkum on Sunday March 13, 2016 @11:29PM (#11280877)
Ok I don't mean this in a mean way but you don't seem to appreciate what CS research is like. It is not about handwriting.
Re:Should All Research Papers Be Free? by Sacho on Monday March 14, 2016 @06:20AM (#11281953)
Filing taxes is not about handwriting either - I guess you missed the analogy. Research isn't about writing good code, but publishing your code is useful(for reproducibility, which is part of research), and if you're embarrassed by your code quality, then you should put some effort into it.
Re:Should All Research Papers Be Free? by return0 on Sunday March 13, 2016 @09:23PM (#11280420)
Absolutely yes. Science isnt supposed to be clean and tidy, but reproducible.
Re:Should All Research Papers Be Free? by bravura on Sunday March 13, 2016 @09:21PM (#11280414)
Yes.

Especially because if you are forced to publish your code, it will be better.

Re:Should All Research Papers Be Free? by nv-vn on Sunday March 13, 2016 @09:23PM (#11280428)
Considering that a mistake in such a program could completely skew results, yeah that seems pretty important.
Re:Should All Research Papers Be Free? by nilkn on Sunday March 13, 2016 @10:46PM (#11280760)
To be honest, I'm surprised anybody even takes research seriously which depends heavily on a custom program whose source code is kept hidden.
Re:Should All Research Papers Be Free? by lacker on Sunday March 13, 2016 @09:24PM (#11280430)
Yes, because if it doesn't even compile then I will call BS on your results.
Re:Should All Research Papers Be Free? by skybrian on Sunday March 13, 2016 @09:41PM (#11280516)
See, that's exactly what you shouldn't do if you want to encourage academics to release the source code.

Academic code sometimes only has been made to work on one machine (the author's). Someone reproducing the results should still have access to the source, even if they have to hack it to work on their system.

Re:Should All Research Papers Be Free? by CompuHacker on Sunday March 13, 2016 @09:27PM (#11280449)
It's better than nothing.
Re:Should All Research Papers Be Free? by themartorana on Sunday March 13, 2016 @09:25PM (#11280439)
I don't know... Did you do it right?

;)

But seriously.

Re:Should All Research Papers Be Free? by dinkumthinkum on Sunday March 13, 2016 @11:27PM (#11280868)
I think you are providing a large barrier unnecessarily. Research projects and publishing a paper is more complicated than that. It is good when the researcher can do that but it is not always practical.

Really, a lot of the comments to this and the downvotes seem to betray a lack of understanding of what cs research is like, which is understandable I guess. CS research is not like creating physics experiments, there are methods, techniques, and algorithms. When there is an experimental result, CS researchers do not treat those as sacrosanct and do have an informed skepticism but we are generally able to determine and discern the bulk is what is important from the content of the paper itself, in most cases.

Yeah, if you look at the experimental results section from papers in SOSP, you could come up with all kinds of complaints probably all day long but there is more to it and the community than is being taken for granted here.