I work with code, which is about as reproducible as it is possible to get - the artifacts I produce are literally just instructions on how to reproduce the work I've done again, and again, and again. And still people come to me with some bug that they've experienced on their machine, that I cannot reproduce on my machine, despite the two environments being as identical as I can possibly make them.

I agree that reproduction in scientific work is important, but it is also apparently impossible in the best possible circumstances. When dealing with physical materials, inexact measurements, margins of error, etc, I think we have to accept that there is no set of instructions that, if followed to a T, will ever ensure perfect replication.

> And still people come to me with some bug that they've experienced on their machine, that I cannot reproduce on my machine

But this is the other way around. Have you ever written a program that doesn't run anywhere except a single machine of yours? Would you release it and advertise it and encourage other people to use it as dependency in their software?

If it only runs on one machine of yours, you don't even know if your code is doing something, or something else in the machine/OS. Or in terms of science, whether the research says something about the world, or just about the research setup.

I think you misunderstand the point of scientific publication here (at least in theory, perhaps less so in practice). The purpose of a paper is typically to say "I have achieved these results in this environment (as far as I can tell)", and encourages reproduction. But the original result is useful in its own right - it tells us that there may be something worth exploring. Yes, it may just be a measurement error (I remember the magic faster than light neutrinos), but if it is exciting enough, and lots of eyes end up looking, then flaws are typically found fairly quickly.

And yes, there are often overly excited press releases that accompany it - the "advertise it and encourage others to us it as a dependency" part of it analogy - but this is typically just noise in the context of scientific research. If that is your main problem with scientific publishing, you may want to be more critical of science journalism instead.

Fwiw, yes of course I've written code that only runs on my machine. I imagine everyone has, typically accidentally. You do it, you realise your mistake, you learn something from it. Which is exactly what we expect from scientific papers that can't be reproduced.

> But the original result is useful in its own right - it tells us that there may be something worth exploring.

I disagree. It shows that when someone writes something in a text editor and publishes it, others can read the words they wrote. That's all it shows, by itself. Just like someone writing something on the web only tells us that a textarea accepts just about any input.

And even if it did show more than that, when someone "explores" it, is the result is more of that, something that might be true, might not be, but "is worth exploring"? Then at what point does falsifiability enter into it? Why not right away? To me it's just another variation of making it someone else's problem, kicking the can down the road.

> if it is exciting enough, and lots of eyes end up looking, then flaws are typically found fairly quickly.

If that was true, there wouldn't even be a replication issue, much less a replication crisis. It's like saying open source means a lot of people look at the code, if it's important enough. Time and time again that's proven wrong, e.g. https://www.zdnet.com/article/open-source-software-security-...

> yes of course I've written code that only runs on my machine. I imagine everyone has

I wouldn't even know how to go about doing that. Can you post something that only runs on one of your machines, and you don't know why? Note I didn't say your machine, I said one machine of yours. Would you publish something that runs on one machine of yours but not a single other one, other than to ask "can anyone tell me why this only runs on this machine"? I doubt it.

> Note I didn't say your machine, I said one machine of yours.

This thread discusses peer replication, this is not even an analogy.

If you can't even replicate it yourself, what makes you think peers could? We are talking about something not being replicated, not even by the original author. The most extreme version would be something that you could only get to run once on the same machine, and never on any other machine.

I think you may be seeing the purpose of these papers differently to me, which may be the cause of this confusion.

The way you're describing a scientific publication is as if it were the end result of the scientific act. To use the software analogy, you're describing publication like a software release: all tests have been performed, all CI workflows have passed, QA have checked everything, and the result is about to be shipped to customers.

But talking to researchers, they see publishing more like making a new branch in a repository. There is no expectation that the code in that branch already be perfect (hence why it might only run on one machine, or not even run at all, because sometimes even something that doesn't work is still worth committing and exploring later).

And just like in software, where you might eventually merge those branches and create a release out of it, in the scientific world you have metastudies or other forms of analysis and literature reviews that attempt to glean a consensus out of what has been published so far. And typically in the scientific world, this is what happens. However, in journalism, this isn't usually what happens, and one person's experimental, "I've only tested this on my machine" research is often treated as equivalent to another person's "release branch" paper evaluating the state of a field and identifying which findings are likely to represent real, universal truths.

Which isn't to say that journalists are the only ones at fault here - universities that evaluate researchers primarily on getting papers into journals, and prestige systems that make it hard to go against conventional wisdom in the field both cause similar problems by conflating different levels of research or adding competing incentives to researchers' work. But I don't think that invalidates the basic idea of published research: to present a found result (or non-really), provide as much information as possible about how to replicate the result again, and then let other people use that information to inform their work. It just requires us to be mindful of how we let that research inform us.

> But talking to researchers, they see publishing more like making a new branch in a repository.

Well some do, others don't. Like the one who wrote the article this is a discussion of.

https://en.wikipedia.org/wiki/Replication_crisis

> Replication is one of the central issues in any empirical science. To confirm results or hypotheses by a repetition procedure is at the basis of any scientific conception. A replication experiment to demonstrate that the same findings can be obtained in any other place by any other researcher is conceived as an operationalization of objectivity. It is the proof that the experiment reflects knowledge that can be separated from the specific circumstances (such as time, place, or persons) under which it was gained.

Or, in short, "one is none". One might turn into more than one, it might not. Until it does, it's not real.

more snippets from the above WP article:

> This experiment was part of a series of three studies that had been widely cited throughout the years, was regularly taught in university courses

> what the community found particularly upsetting was that many of the flawed procedures and statistical tools used in Bem’s studies were part of common research practice in psychology.

> alarmingly low replication rates (11-20%) of landmark findings in preclinical oncological research

> A 2019 study in Scientific Data estimated with 95% confidence that of 1,989 articles on water resources and management published in 2017, study results might be reproduced for only 0.6% to 6.8%, even if each of these articles were to provide sufficient information that allowed for replication

I'm not saying it couldn't be fine to just publish things because they "could be interesting". But the overall situation seems like quite the dumpster fire to me. As does software, FWIW.

You just described the majority of scientific papers. A "working set of instructions" is not really feasible in most cases. You can't include every piece of hard- and software required to replicate your own setup.

Then don't call it science, since it doesn't contribute anything to the body of human knowledge.

I think it's fascinating that we can at the same time hold things like "one is none" to be true, or that you should write tests first, but with science we already got so used to a lack of discipline that we just declare it fine.

It's not hard to not climb a tower you can't get down from. It's the default, actually. You start with something small where you can describe everything that goes into replicating it. Then you replicate it yourself, based on your own instructions. Before that, you don't bother anyone else with it. Once that is done, and others can replicate as well, it "actually exists".

And if that means the majority of stuff has to be thrown out, I'd suggest doing that sooner rather than later, instead of just accumulating scientific debt.

Imagine two scientists, Bob and Alice. Bob has spent the last 5 years examining a theory thoroughly. Now he can explain down to the last detail why the theory does not hold water, and why generations of researchers have been wrong about the issue. Unfortunately, he cannot offer an alternative, and nobody else can follow his long winded arguments anyway.

Meanwhile, Alice has spent the last 5 years making the best possible use of the flawed theory, and published a lot of original research. Sure, many of her publications are rubbish, but a few contain interesting results. Contrary to Bob, Alice can show actual results and has publications.

Who do you believe will remain in academia? And, according to public perception, will seem more like an actual scientist?

I disagree that Bob doesn't produce actual results, or that something that is mostly rubbish, but partly "interesting" is an actual result. We know the current incentives are all sorts of broken, across the board. Goodhart's law and all that. To me the question isn't who remains in academia given the current broken model, but who would remain in academia in one that isn't as broken.

To put a point on it, if public distrust of science becomes big enough, it all can go away before you can say "cultural revolution" or "fascist strongman". Then there'd be no more academia, and its shell would be inhabited by party members, so to speak. I'd gladly sacrifice the ability of Alice and others like her to live off producing "mostly rubbish" to at least have a chance to save science itself.

Then Bob has failed.

Academic science isn’t just the doing science part but the articulation and presentation of your work to the broader community. If Bob knows this space so well, he should be able to clearly communicate the issue and, ideally, present an easily understandable counter example to the existing theory.

Technical folks undervalue presentation when writing articles and presenting at conferences. The burden of proof is on the presenter, and, unless there’s some incredible demonstration at the end, most researchers won’t have the time or attention to slog through your mess of a paper to decipher it. There’s only so much time in the day and too many papers to read.

In my experience, the best researchers are also the best presenters. I’ve been to great talks out of my domain that I left feeling like I understood the importance of their work despite not understanding the details. I’ve also seen many talks in my field that I thought were awful because the presentation was convoluted or they didn’t motivate the importance of their problem / why their work addressed it

This is a very simplistic view. Why do believe QC departments exist? Even in an industrial setting, companies make the same thing at the same place on the same equipment after sometimes years of process optimisation of well understood technology. This is essentially a best case scenario and still results fail to reproduce. How are scientists who work at the cutting edge of technology with much smaller budgets supposed to give instructions that can be easily reproduced on first go? Moreover how are they supposed to easily reproduce other results?

That is not to say that scientist should not document the process to their best ability so it can be reproduced in principle. I'm just arguing that it is impossible to easily reproduce other people's results. Again when chemical/manufacturing companies open another location they often spend months to years to make the process work in the new factory.

> companies make the same thing at the same place on the same equipment after sometimes years of process optimisation of well understood technology. This is essentially a best case scenario and still results fail to reproduce.

We're not talking about 1 of 10 reproduction attempts failing, we're talking about 100%. And no, companies don't time and time again try to reproduce something that has never been reproduced and fail, to then try again, endlessly. That's just not a thing.

> it is impossible to easily reproduce other people's results

We're also not talking about "easily" reproducing something, but at all. And in principle doesn't cut it, it needs to be reproduced in practice.

Sounds like a problem worth solving.

You should.