There are some incentives to make work reproducible. If you publish a method implemented in software, then making that software easy to use, well documented, with example data and respond to questions/issues etc it greatly enhances the chance other people will use it and cite it. Things have been getting much better in the last 5 years.
Preprints also help - I saw a preprint the other day where people complained on twitter there was no methods section and no code. The authors responded. They realise that potential reviewers may be the ones making these comments or at least see the comments. This talks to your point about not treating a paper as a single point in time, but an ongoing process.
I think things are improving. I am now seeing papers publish Jupyter notebooks in python or R scripts to reproduce all the diagrams and analyses, along with curated data that just plugs in. In general this actually works (although sometimes crucial bits of data are missing, and unlike an error in the paper, these are not necessarily fixed).
Snakemake is a parallel build tool designed for data instead of software as its main use case.
You're right. I use laziness out of frustration, but in reality I know it's hard for me to make my own work fully documented and reproducible.
A large part of the situation is cultural. Software engineers are born into a world of version control, unit tests, documentation, managing complexity, reproducibility, debugability, etc (and we still struggle with it). That sort of culture and associated tools are missing from data science.
Think of a research lab as a company that gets paid per prototype and then has to market the concept for the next prototype in an infinite loop. If you can't package up what you're doing into a sequence of small prototypes then you're not getting paid.
You can't expect the results to be reproductible 20 years later otherways.
Speaking of, I haven't played with R - what are its standard methods for handling dependencies? I'm particularly enamored of the pip and npm way of doing it, where you create a version-controlled artifact (requirements.txt and packages.json, respectively) that defines your dependencies. Does R not have a similar system, or do people just not use it?
I'm a bit bitter about the whole "writing reproducible code in R", as I'm currently wasting a lot of time trying to get R code I wrote at the start of my PhD to run again now I'm writing up.
also, labs keep the data around, and sometimes use it again for novel posthoc analyses, even years later. in fact that's what I'm doing now!
I first started thinking about this problem after I heard this talk by Arfon Smith [1]. It contextualizes reproducibility tangentially in discussions about viewing modern research as a heterogeneous (nodes can represent papers, experiments, software, authors, etc.) dependency network.
[1] https://speakerdeck.com/arfon/academias-biggest-blind-spot
OTOH, it would be awesome for me, as I've been doing consulting on exactly this for 15+ years :)
How do you do science if you can't repeat experiments?
"How do you do science if you can't repeat experiments?"
Well, the way we've been doing it for 300 years? It's not like the flawed processes of the past didn't yield anything. To put it in operations research terms: science doesn't work like hill climbing, it's more like a stochastic optimization process without a (well defined) halting condition. Again, I'm not saying we can't and shouldn't strive to improve, but the first year grad student level huff puffery in this thread is just completely detached from reality. 'Science' as you were explained the concept in high school is only a high-level abstraction of the concept, it's not how actual things are or get done in the real world.
personally I find the worriers overstating the problem of reproducibility. lots of results are reproduced all the time, but in the course of testing other hypotheses.
if a result depends on exact replication of methods it isn't that robust an effect, and just might be a trivial one.
That's the tool, and that page cites the paper.
Who is paying for non-reproducible papers? What’s the point?
https://en.wikipedia.org/wiki/Reproducibility#Reproducible_r...
The term reproducible research refers to the idea that the ultimate product of academic research is the paper along with the laboratory notebooks [12] and full computational environment used to produce the results in the paper such as the code, data, etc. that can be used to reproduce the results and create new work based on the research.[13][14][15][16][17] Typical examples of reproducible research comprise compendia of data, code and text files, often organised around an R Markdown source document[18] or a Jupyter notebook.[19]
> abstract numerical disciplines
Could you give an example of a non-numerical science?
"Could you give an example of a non-numerical science?"
This is the point you're probably going to argue about what is or is not 'science', so let me use 'scholarship' instead from the fields I work in: law, environmental science and geography. There is a lot of research that is not purely numerical in nature. Note that this doesn't mean that it doesn't use numbers; you misread what I wrote - I didn't say 'numerical disciplines', I said 'abstract numerical disciplines', by which I meant disciplines that require at least some judgement of qualitative properties or inexact measurements of things along the way, somewhere.
No, but I had spent a whole year at university trying to replicate experiments in physics lab. I was studying CS but they had too many physics teachers so we had many of the same courses :)
The equipment was abused by decades of students, you had 45 minutes to do the experiment, and the results usually weren't even the correct order of magnitude :) It was stuff like measuring speed of sound, atmospheric pressure, gravity constant, etc.
Still - at least I knew it's the problem with equipment or with me, not with some obscure details that weren't mentioned and that I can't replicate.
> My point is that there are a lot of aspects to most research that aren't easy (or even at all) to replicate
And my point is that it makes publishing everything that can be adjusted that much more important.
Another question - when there's economic and proffesional incentive to fudge the numbers, and no way to check if the numbers were fudged - how do you trust the results?
I wouldn't.
If a paper is very novel and impactful but difficult to reproduce, it doesn't seem to matter much for citation counts as long as people just believe it to be true.
And yet there's no revolution among funding sources to send money away from those and instead towards researchers who spend more time on it.
I’d assume people are paying for it who want high quality research.
This is why everything should have CI, and one should ask at least one person to set up the system from scratch using just the documentation.
I found a paper that deleted (one at a time) each gene in yeast, to see which ones were "absolutely required". They published the list. For each "novel discovery of a required gene", I was able to show the gene overlapped (yes, genes overlap) with an already known required gene, and so, much of the paper's "novel discovery" section came into doubt.
It's nice because genes have precise coordinates and range intersection is cheap to compute.
Big governments need to fund it and solve this problem once and for all. At the moment most grant procedures require something more concrete than just "solve the reproducibility problem", though.
One major problem is that there's not much real incentive to make your work reproducible. Money granting organizations favor researchers breaking new and exciting ground, not those rehashing an already published method. Publishers don't require reproducible methods, and reviewers don't have the time, desire, nor expertise to do an in-depth methods review.
Wet lab experiments are 1-2 orders of magnitude more expensive and difficult to reproduce, that's true, but we're not even getting the basics right!