News for hackers. Stuff that matters.
Re:Plan to replicate 50 high-impact cancer papers shrinks to... by buchanae on Monday August 06, 2018 @10:01PM (#17703363)
I have been working on reproducing computational biology papers from the cancer field lately. I am very frustrated. When the inputs and outputs are machine-readable data, there's no excuse for not making your work reproducible, in my opinion. Often the problem is plain laziness and disorganization.

One major problem is that there's not much real incentive to make your work reproducible. Money granting organizations favor researchers breaking new and exciting ground, not those rehashing an already published method. Publishers don't require reproducible methods, and reviewers don't have the time, desire, nor expertise to do an in-depth methods review.

Wet lab experiments are 1-2 orders of magnitude more expensive and difficult to reproduce, that's true, but we're not even getting the basics right!

Re:Plan to replicate 50 high-impact cancer papers shrinks to... by Gatsky on Tuesday August 07, 2018 @02:11AM (#17704202)
Yeah there is no excuse for comp bio work really, but I don't think you should cast aspersions of laziness. By far the least lazy people I know are scientists. Publishing a paper is an incredible amount of work often under less than ideal conditions. The authors are usually exhausted. That isn't an excuse, but it is a factor nonetheless.

There are some incentives to make work reproducible. If you publish a method implemented in software, then making that software easy to use, well documented, with example data and respond to questions/issues etc it greatly enhances the chance other people will use it and cite it. Things have been getting much better in the last 5 years.

Preprints also help - I saw a preprint the other day where people complained on twitter there was no methods section and no code. The authors responded. They realise that potential reviewers may be the ones making these comments or at least see the comments. This talks to your point about not treating a paper as a single point in time, but an ongoing process.

I think things are improving. I am now seeing papers publish Jupyter notebooks in python or R scripts to reproduce all the diagrams and analyses, along with curated data that just plugs in. In general this actually works (although sometimes crucial bits of data are missing, and unlike an error in the paper, these are not necessarily fixed).

Re:Plan to replicate 50 high-impact cancer papers shrinks to... by rspeer on Tuesday August 07, 2018 @02:47AM (#17704336)
I'd probably encounter the bad kind of comp bio if I actually worked in that field, but a tool made by computational biologists for reproducible research has also been very useful in my work: the build tool Snakemake [1].

Snakemake is a parallel build tool designed for data instead of software as its main use case.

[1] http://snakemake.readthedocs.io/en/stable/

Re:Plan to replicate 50 high-impact cancer papers shrinks to... by Q6T46nT668w6i3m on Tuesday August 07, 2018 @09:41AM (#17706463)
It should also be said major parts of the Python scientific ecosystem was written by computational biologists!
Re:Plan to replicate 50 high-impact cancer papers shrinks to... by buchanae on Tuesday August 07, 2018 @10:30AM (#17706943)
> I don't think you should cast aspersions of laziness

You're right. I use laziness out of frustration, but in reality I know it's hard for me to make my own work fully documented and reproducible.

A large part of the situation is cultural. Software engineers are born into a world of version control, unit tests, documentation, managing complexity, reproducibility, debugability, etc (and we still struggle with it). That sort of culture and associated tools are missing from data science.

Re:Plan to replicate 50 high-impact cancer papers shrinks to... by Q6T46nT668w6i3m on Tuesday August 07, 2018 @09:40AM (#17706457)
That would be awesome and I think everyone would like to do this but funding for that barely exists. If you’re in the United States, complain to NIH, NSF, and especially your rep.
Re:Plan to replicate 50 high-impact cancer papers shrinks to... by buchanae on Monday August 06, 2018 @10:03PM (#17703381)
And another thing! Why should a project be done when it gets published?! What kind of software project would make one release and call it good forever? Nonsense.
Re:Plan to replicate 50 high-impact cancer papers shrinks to... by fmap on Tuesday August 07, 2018 @01:33AM (#17704100)
In an ideal world you would be right, but that's not how the funding structures for academic research are set up.

Think of a research lab as a company that gets paid per prototype and then has to market the concept for the next prototype in an infinite loop. If you can't package up what you're doing into a sequence of small prototypes then you're not getting paid.

Re:Plan to replicate 50 high-impact cancer papers shrinks to... by ajuc on Tuesday August 07, 2018 @02:26AM (#17704245)
The requirement for publishing in a scientific journalshould be opensourcing all the code used.

You can't expect the results to be reproductible 20 years later otherways.

Re:Plan to replicate 50 high-impact cancer papers shrinks to... by skummetmaelk on Tuesday August 07, 2018 @04:27AM (#17704723)
20 year old code that has not been maintained most likely would not run on a modern system anyway, unless it is extremely simple.
Re:Plan to replicate 50 high-impact cancer papers shrinks to... by azernik on Tuesday August 07, 2018 @04:39AM (#17704771)
Research labs operate on a very slow tech upgrade cycle anyway; since code is handed down from assistant to post-doc to grad student, complete rewrites would take up a significant fraction of a person's time at any given lab, and so codebases are often as long-lived as the labs in which they live. We're talking decades-old FORTRAN here. Running twenty-year-old software is a barrier for some labs, but not all.
Re:Plan to replicate 50 high-impact cancer papers shrinks to... by _Wintermute on Tuesday August 07, 2018 @01:05PM (#17708407)
My experience is the opposite, that a lot of labs are running R code that won't work 6 months later, and no one actually recorded the package version numbers that were used.
Re:Plan to replicate 50 high-impact cancer papers shrinks to... by azernik on Tuesday August 07, 2018 @05:58PM (#17711241)
TBF, the experience I have is with physics and mechanical/civil engineering labs, where there historically hasn't been much of a reliance on R. And in any case said experience is several years out of date.

Speaking of, I haven't played with R - what are its standard methods for handling dependencies? I'm particularly enamored of the pip and npm way of doing it, where you create a version-controlled artifact (requirements.txt and packages.json, respectively) that defines your dependencies. Does R not have a similar system, or do people just not use it?

Re:Plan to replicate 50 high-impact cancer papers shrinks to... by _Wintermute on Wednesday August 08, 2018 @03:23AM (#17713827)
R isn't fantastic for handling dependencies. If your code is bundled up as a package then you can specify version numbers for your dependencies, but I don't know of any equivalent to `pip freeze` to actually list these. Installing anything other than the latest version of a package is a bit of a pain, and setting up environments for separate projects is pretty much unheard of.

I'm a bit bitter about the whole "writing reproducible code in R", as I'm currently wasting a lot of time trying to get R code I wrote at the start of my PhD to run again now I'm writing up.

Re:Plan to replicate 50 high-impact cancer papers shrinks to... by nonbel on Wednesday August 08, 2018 @07:33AM (#17714965)
Re:Plan to replicate 50 high-impact cancer papers shrinks to... by ajuc on Tuesday August 07, 2018 @04:40AM (#17704773)
It can always be ported. Without the code you're stuck.
Re:Plan to replicate 50 high-impact cancer papers shrinks to... by a_136_chiffa on Friday August 10, 2018 @12:45PM (#17734876)
One where you don't get sustained funding to maintain it. In compbio even major resources, known to everyone in the domain only have funding from one two-year grant to another.
Re:Plan to replicate 50 high-impact cancer papers shrinks to... by your-nanny on Tuesday August 07, 2018 @09:05AM (#17706152)
I don't really understand what you are suggesting should be done? data-mining?

also, labs keep the data around, and sometimes use it again for novel posthoc analyses, even years later. in fact that's what I'm doing now!

Re:Plan to replicate 50 high-impact cancer papers shrinks to... by roel_v on Tuesday August 07, 2018 @01:58AM (#17704163)
Who pays for that?
Re:Plan to replicate 50 high-impact cancer papers shrinks to... by wjn0 on Monday August 06, 2018 @10:28PM (#17703496)
Yup, it is very frustrating - and discouraging. There are some tools designed for replication - Snakemake, for one, in python, which seems to work pretty well. But, I think it adds unnecessary dependencies and complexity. I support using basic using basic UNIX tools for reproducibility, when possible: git to make all code publicly accessible, and make to reproduce all experiments and figures. I've implemented it in 2 comp bio projects (one which relies on publicly accessible data, and one which doesn't). In both cases it massively improved my workflow. It has the added benefit of being language-independent: when I have to blend R and python, it works seamlessly.

I first started thinking about this problem after I heard this talk by Arfon Smith [1]. It contextualizes reproducibility tangentially in discussions about viewing modern research as a heterogeneous (nodes can represent papers, experiments, software, authors, etc.) dependency network.

[1] https://speakerdeck.com/arfon/academias-biggest-blind-spot

Re:Plan to replicate 50 high-impact cancer papers shrinks to... by benhamner on Tuesday August 07, 2018 @12:00AM (#17703773)
Strongly agree with this. We're aiming to make Kaggle Kernels/Datasets as reproducible as possible while being easy to use, with versioned data, versioned code, and versioned compute environments (the latter through Docker containers). https://www.kaggle.com/kernels
Re:Plan to replicate 50 high-impact cancer papers shrinks to... by yellow_postit on Monday August 06, 2018 @10:27PM (#17703490)
One of the areas paid journals could make the case for their value and relevance would be in demonstrating reproducability.
Re:Plan to replicate 50 high-impact cancer papers shrinks to... by qudat on Tuesday August 07, 2018 @12:02AM (#17703783)
Requiring reproducibility would radically change the academic landscape for scientists. I'm sure in the end it would be for the greater good but I think most career researchers in the US would hate this kind of change.
Re:Plan to replicate 50 high-impact cancer papers shrinks to... by roel_v on Tuesday August 07, 2018 @02:04AM (#17704179)
It would require either all researchers to be 25% software engineers, or to add 30-50% headcount by adding SE's. It would be a radical change in methods and funding. There is massive institutional pushback to moving in this direction, as people already feel too much is done in/by computers. To make any research to be publishable depend on software engineering technicalities...

OTOH, it would be awesome for me, as I've been doing consulting on exactly this for 15+ years :)

Re:Plan to replicate 50 high-impact cancer papers shrinks to... by ajuc on Tuesday August 07, 2018 @02:28AM (#17704252)
Is it better to get the correct answers 25% slower, or incorrect answers at 100% speed?

How do you do science if you can't repeat experiments?

Re:Plan to replicate 50 high-impact cancer papers shrinks to... by roel_v on Tuesday August 07, 2018 @08:12AM (#17705741)
shrug What does 'the correct answer' even mean for any real world problem? There is so much we don't know. I'm not saying we should just make up papers and call it good, but there is a gradient between 'making stuff up' and 'doing everything according to a hypothetical ideal methodology all the time'. It's a matter of opinion how much not having a 100% repeatable experiment for each paper moves us towards the left of that gradient. It would be an improvement, yes, but I would put it more in the 'moves us from 75 to 80' ballpark than in the 'moves us from 20 to 100' that most people in this thread seem to believe.

"How do you do science if you can't repeat experiments?"

Well, the way we've been doing it for 300 years? It's not like the flawed processes of the past didn't yield anything. To put it in operations research terms: science doesn't work like hill climbing, it's more like a stochastic optimization process without a (well defined) halting condition. Again, I'm not saying we can't and shouldn't strive to improve, but the first year grad student level huff puffery in this thread is just completely detached from reality. 'Science' as you were explained the concept in high school is only a high-level abstraction of the concept, it's not how actual things are or get done in the real world.

Re:Plan to replicate 50 high-impact cancer papers shrinks to... by your-nanny on Tuesday August 07, 2018 @09:12AM (#17706215)
"first year grad student level huff puffery"

personally I find the worriers overstating the problem of reproducibility. lots of results are reproduced all the time, but in the course of testing other hypotheses.

if a result depends on exact replication of methods it isn't that robust an effect, and just might be a trivial one.

Re:Plan to replicate 50 high-impact cancer papers shrinks to... by ahartmetz on Tuesday August 07, 2018 @03:42AM (#17704542)
I have worked on experimental high-energy physics with computers and had the same idea - physicists should just hire software engineers. Large amounts of physicist time are spent cobbling together, and suffering from, shoddy software. At least we had public repositories with version numbers, and datasets were shared and uniquely named.
Re:Plan to replicate 50 high-impact cancer papers shrinks to... by your-nanny on Tuesday August 07, 2018 @09:12AM (#17706220)
that's my job!
Re:Plan to replicate 50 high-impact cancer papers shrinks to... by fadys on Monday August 06, 2018 @11:20PM (#17703650)
Can you point to some computational biology papers from the cancer field that you're reproducing? I'd love to give it a go myself. If you don't mind, can you also contact me (email is in my profile)? Maybe we can collaborate.
Re:Plan to replicate 50 high-impact cancer papers shrinks to... by buchanae on Tuesday August 07, 2018 @10:34AM (#17706977)
Here's one I find tricky: https://bioconductor.org/packages/release/bioc/html/viper.ht...

That's the tool, and that page cites the paper.

Re:Plan to replicate 50 high-impact cancer papers shrinks to... by drb91 on Monday August 06, 2018 @11:22PM (#17703661)
> One major problem is that there's not much real incentive to make your work reproducible.

Who is paying for non-reproducible papers? What’s the point?

Re:Plan to replicate 50 high-impact cancer papers shrinks to... by perl4ever on Tuesday August 07, 2018 @01:29PM (#17708651)
The Journal of Irreproducible Results[1], of course!

[1]http://www.jir.com/

Re:Plan to replicate 50 high-impact cancer papers shrinks to... by roel_v on Tuesday August 07, 2018 @02:07AM (#17704189)
No, the question is: who is paying for reproducible ones? What does it even mean, apart from in some abstract numerical disciplines?
Re:Plan to replicate 50 high-impact cancer papers shrinks to... by ajuc on Tuesday August 07, 2018 @02:34AM (#17704269)
> What does it even mean

https://en.wikipedia.org/wiki/Reproducibility#Reproducible_r...

The term reproducible research refers to the idea that the ultimate product of academic research is the paper along with the laboratory notebooks [12] and full computational environment used to produce the results in the paper such as the code, data, etc. that can be used to reproduce the results and create new work based on the research.[13][14][15][16][17] Typical examples of reproducible research comprise compendia of data, code and text files, often organised around an R Markdown source document[18] or a Jupyter notebook.[19]

> abstract numerical disciplines

Could you give an example of a non-numerical science?

Re:Plan to replicate 50 high-impact cancer papers shrinks to... by roel_v on Tuesday August 07, 2018 @07:55AM (#17705630)
Yes of course I know what the Wikipedia definition is. Have you ever tried to 'replicate' a paper? Let's say someone did a survey of something. How are you going to 'replicate' that? Or used a particular piece of equipment, or did some measurements in a field? Sure, it's easy to restrict yourself to the data files that go into some software, and say 'oh I used the same data set and software version, my research is 'replicatable'.' My point is that there are a lot of aspects to most research that aren't easy (or even at all) to replicate, even if one is put in all the effort required to document every minute circumstance. There is more to 'science' than CS algos and lab bench biomed.

"Could you give an example of a non-numerical science?"

This is the point you're probably going to argue about what is or is not 'science', so let me use 'scholarship' instead from the fields I work in: law, environmental science and geography. There is a lot of research that is not purely numerical in nature. Note that this doesn't mean that it doesn't use numbers; you misread what I wrote - I didn't say 'numerical disciplines', I said 'abstract numerical disciplines', by which I meant disciplines that require at least some judgement of qualitative properties or inexact measurements of things along the way, somewhere.

Re:Plan to replicate 50 high-impact cancer papers shrinks to... by ajuc on Tuesday August 07, 2018 @11:15AM (#17707353)
> Have you ever tried to 'replicate' a paper?

No, but I had spent a whole year at university trying to replicate experiments in physics lab. I was studying CS but they had too many physics teachers so we had many of the same courses :)

The equipment was abused by decades of students, you had 45 minutes to do the experiment, and the results usually weren't even the correct order of magnitude :) It was stuff like measuring speed of sound, atmospheric pressure, gravity constant, etc.

Still - at least I knew it's the problem with equipment or with me, not with some obscure details that weren't mentioned and that I can't replicate.

> My point is that there are a lot of aspects to most research that aren't easy (or even at all) to replicate

And my point is that it makes publishing everything that can be adjusted that much more important.

Another question - when there's economic and proffesional incentive to fudge the numbers, and no way to check if the numbers were fudged - how do you trust the results?

I wouldn't.

Re:Plan to replicate 50 high-impact cancer papers shrinks to... by drb91 on Tuesday August 07, 2018 @11:18AM (#17707376)
Well, if someone else can’t reproduce the research, what’s the value of it? Any knowledge worth finding can be found many times.
Re:Plan to replicate 50 high-impact cancer papers shrinks to... by esrauch on Tuesday August 07, 2018 @03:37AM (#17704518)
I think you too his question too literally: I think he meant "what does it even mean pragmatically and why do you expect funding to go to less novel but more easily reproducible results instead of more novel and harder to reproduce papers?"
Re:Plan to replicate 50 high-impact cancer papers shrinks to... by drb91 on Tuesday August 07, 2018 @11:19AM (#17707382)
Well, reproduced research is inherently more valuable than novel but unsupported research. Otherwise why fund research at all? Just hire alchemists or poets, at least you’ll be entertained.
Re:Plan to replicate 50 high-impact cancer papers shrinks to... by esrauch on Wednesday August 08, 2018 @02:40AM (#17713665)
The current state of research seems to reflect that there is a balance: almost none of the most cited papers go really far out of theirs way to be trivially reproducible, even in computing where doing so is unusually much easier.

If a paper is very novel and impactful but difficult to reproduce, it doesn't seem to matter much for citation counts as long as people just believe it to be true.

And yet there's no revolution among funding sources to send money away from those and instead towards researchers who spend more time on it.

Re:Plan to replicate 50 high-impact cancer papers shrinks to... by thecatspaw on Tuesday August 07, 2018 @05:03AM (#17704883)
funding should go to more reproducible studies because those are the ones that are usefull. What use is a study saying that X will cure cancer Y if noone can reproduce that and verify?
Re:Plan to replicate 50 high-impact cancer papers shrinks to... by drb91 on Tuesday August 07, 2018 @11:14AM (#17707333)
Well, it means you have a higher confidence level, which is pretty much just standard induction.

I’d assume people are paying for it who want high quality research.

Re:Plan to replicate 50 high-impact cancer papers shrinks to... by notimetorelax on Tuesday August 07, 2018 @02:05AM (#17704180)
I think it’s the same problem as trying to document how to install or use new software. Unless you redo the steps yourself or somebody does that for you, you won’t know which assumptions you forgot to document. Since no-one retries the study right after you finished it, there out to be gaps in the documentation.
Re:Plan to replicate 50 high-impact cancer papers shrinks to... by jononor on Tuesday August 07, 2018 @06:59AM (#17705366)
Yes. "If it ain't tested, it is broken" applies just as much to documentation.

This is why everything should have CI, and one should ask at least one person to set up the system from scratch using just the documentation.

Re:Plan to replicate 50 high-impact cancer papers shrinks to... by dekhn on Tuesday August 07, 2018 @09:26AM (#17706332)
by the way, when inputs and outputs are clearly available, it's amazing how easy it to eliminate papers from consideration.

I found a paper that deleted (one at a time) each gene in yeast, to see which ones were "absolutely required". They published the list. For each "novel discovery of a required gene", I was able to show the gene overlapped (yes, genes overlap) with an already known required gene, and so, much of the paper's "novel discovery" section came into doubt.

It's nice because genes have precise coordinates and range intersection is cheap to compute.

Re:Plan to replicate 50 high-impact cancer papers shrinks to... by Q6T46nT668w6i3m on Tuesday August 07, 2018 @09:38AM (#17706437)
Many of the popular journals in cancer biology have strict data retention policies. If that was the case in your example, you should probably contact the editor but also appreciate that mistakes do happen and most people are not cheating the system.
Re:Plan to replicate 50 high-impact cancer papers shrinks to... by cup-of-tea on Tuesday August 07, 2018 @08:06AM (#17705708)
It's quite difficult even for trained programmers to make their pipelines reproducibile. Just look at how much effort goes into build pipelines and the like. There isn't any good tooling for doing it so people build their own systems ad-hoc which they never document, that is if they use a system at all.

Big governments need to fund it and solve this problem once and for all. At the moment most grant procedures require something more concrete than just "solve the reproducibility problem", though.