Just because a study doesn't replicate, doesn't make it false. This is especially true in medicine where the potential global subjects/population are very diverse. You can do a small study that suggests further research based on a small sample size, or even a case study. The next study might have a conflicting finding, but that doesn't make the first one false - rather a step in the overall process of gaining new information.
In lieu of actual fraud or a methodological mistake that wasn't represented/caught in peer review, it's still extremely difficult to control for all possible sources of variation. That's especially true as you go further "up the stack" from math -> physics -> chem -> bio -> psych -> social. It is absolutely possible to honestly conduct a very high quality experiment with a real finding, but fail to account for something like "on the way here, 80% of participants encountered a frustrating traffic jam."
Their finding could be true for people who just encountered a traffic jam, and lack of replication would be due to an unsuccessful generalization from what they found.
In what sense do they not? On the assumption that there can be other "worlds" for which math, but not physics, holds?
A geology that isn't anchored to our physical reality seems intrinsically invalid.
But it also doesn’t make it not false. It makes the null hypothesis more likely to be true.
The other is the introduction or loss of critical cofactors or confounders that radically change environment and context.
Think of experiments of certain types before and after COVID-19.
This is a subtle point, but truth or falsity isn't really the issue. The problem with a non-replicable study is that the rest of science can't build on it. You can't build your PhD on top of a handful of studies that turn out to be non-replicable, and so on. It is true you can't build science on outright false statements, either, but true statements that aren't adequately reproducible are also not a solid enough foundation. That may seem counterintuitive, but it comes down to this truth not being binary; even if a study comes to a nominally true conclusion it still matters if it didn't do it via the correct method, or is somehow otherwise deficient in the path it took to get there. Studies are more than just the headline result in the abstract.
But the whole process of science right now is based on building up over time. How could it not be? It has to be, of course. But non-replicable studies mean that the things you're trying to build on them are non-replicable too. It doesn't take all that much before you're just operating in a realm of flights of fancy where you may "feel" like you're on solid ground because of all the Science you're sitting on top of, but it's all just so much air.
However, it is also true that non-replicability is a signal of falsity, and that is simply due to the fact that the vast, vast, vast, exponential majority of all possible hypotheses are false. As is another subtle point, a scientist engaging in science properly should probably not come to that conclusion and may not want to change their priors about something because of a non-reproducible study very much, but externally, from the generalized perspective of "what is true and is not true" where science is merely one particularly useful tool and not the final arbiter, I may be justified in taking non-replicable studies and updating my priors to increase the odds of the hypothesis being false. After all, at the very least, a non-replicable study tends to put an upper bound on the ability of the hypothesis to be true (e.g., if someone studies whether or not substance X kills bacteria Y, and it turns out not to reproduce very well, the lack of reproducibility does fairly strongly establish it can't be that lethal).
This isn't true at all. You could have future experiments set out to prove the opposite, or to dig into what novel and confounding variables are at play causing conflicting results. The fact that you have studies that inconsistently replicate is actually a sign of possible new knowledge if you can find out which undiscovered variable is causing the conflicts. Especially in medicine, you can have a case study that leads to future studies even though that specific case may not be able to be consistently replicated in other n=1 populations.
"e.g., if someone studies whether or not substance X kills bacteria Y, and it turns out not to reproduce very well, the lack of reproducibility does fairly strongly establish it can't be that lethal"
Or there's another factor that hasn't been discovered, such as back in the day before understanding gram positive and negative bacteria was even a thing. If you don't know the subtypes exist, you would see inconsistent results as that is an undiscovered variable that you can't possible account for until it's discovered. Once discovered and controlled for, the substance could be very lethal.
Scientists do not consider a study to be more than an observation. What matters to scientists is the totality of the evidence.
There are several chemistry youtubers who have failed to "reproduce" papers they are working from. Does that mean chemistry is a farce and doesn't work? No, it means some chemist at some point in history failed to write something down, mostly because they didn't even know or realize it mattered. Science is incredibly difficult and we can only hope to be okay at it.
I alter Ionnides's conclusion to be instead: "Roughly 50% of papers in quantitative biological sciences contain at least one error serious enough to invalidate the conclusion" and "Roughly 75% of really interesting papers are missing at least one load-bearing method detail that reproducers must figure out on their own" (my own personal observations of the literature are consistent with these rates; I was always flabbergasted at people who just took Figure 3 as correct).
https://grants.nih.gov/grants/guide/notice-files/NOT-OD-24-1...
Unfortunately, not all NIH institutes understand how to evaluate and moderate this key new policy. Oddly enough the peer reviewers do NOT have access to DMS plans as of this year.
The NIH DMS mandates are about the data generated by an award.
You're talking about (almost certainly) fraudsters denying they committed fraud. The vast majority of non-replicable results have nothing to do with these types of errors, purposeful or not.
Fraud requires intent; it's a word that describes what happened, but also the motivations of the people involved. Incompetence doesn't assume any intent at all; it's merely a description of the (lack of) ability of the people involved.
Incompetent people can certainly commit fraud (perhaps to try to cover up their incompetence), but that's by no means required.
> ...insist they just made honest mistakes
If they're lying about that, it's fraud; they're either covering up their unrealized incompetence with fraud, or trying to cover up their intended fraud with protestations of mere incompetence. If they really did make honest mistakes, then it's just garden-variety incompetence. (Or just... mistakes. To me, incompetence is when someone consistently makes mistakes often. One-time or few-time mistakes are just things that happen to people, no matter how good the are at what they do.)
Maybe someone was incompetent but also knew they were cutting corners. Should they get a pass because they claim they didn't mean to do it? We should hold people accountable regardless of intent.
Not sure what rare means in this context. The more important research is, the more likely there is fraud involved. So in terms of size of impact, it's probably very common.
And then if you combine this with poorly done, non repeatable, or inconclusive research being parroted as a discovery...You end up with quite a bit of BS research.
Part of the reason this paper was impactful is that it was short and punchy, took aim at all of medicine rather than a smaller subfield, and didn't require as much mathematical understanding as other papers.
I knew some of the big names that were hit by the replication crisis. And before that I spent some time trying to talk to psychology researchers at a top school about the problems with statistical testing. But they had limited knowledge of stats and didn't want to go out on a limb when everyone else in the field seemed okay with the status quo. A paper like this can be read by everyone and makes a forceful argument.
> It then try to generalize to "research" and avoid this very narrow support to the claim
This is a good point. The methods in medicine and the social sciences are especially weak and prone to these sorts of criticisms. In the physical sciences, often you can run enough iterations of the experiment to overwhelm any prior.
> You have limitations in tech and tools you use. You might miss something, didn't have access to more information that could change the outcome. That is why research is a process.
I totally agree. Science is basically a control system, or a root finding algorithm, or gradient descent. At any time t there is a gap between the best known science and the truth. But the point is that science converges to the truth over time, whereas no other alternative does.
If the foundational assumptions are wrong or impossible to challenge, time t can extend indefinitely. Additionally, it is surely possible that whole sections of science is waylaid and diverges from truth on account of funding, legislation, big personalities, etc.
There's actually quite a big assumption in here -- namely, that the truth is constant over time and throughout space. (This can possibly be weakened slightly, but that's the gist.)
I personally think the laws of physics probably are unchanging, and I certainly hope they are (because that means the scientific method converges). But whether they actually are is not only unknown, but a question that cannot be answered by empirical science.
But from a technical perspective, the math is perfectly capable of describing theoretical universes where the laws of physics change in time or space. For example, where physical constants are smoothly evolving or manifolds that don't look the same in all directions.
The math can tell us what observations we'd expect in those situations, and so far we haven't observed anything to indicate we should relax those assumptions.
If we did observe that the laws of physics were changing, that observation would be considered science in the traditional sense. So it's not that "truth" is shifting, it's that "truth" is a family of equations indexed by some parameter rather than a single equation. That's a similar flavor to the jump from Newtonian physics to relativity.
If you're interested in this topic, a relevant key word is "cosmological principle": https://en.wikipedia.org/wiki/Cosmological_principle
The current models include these assumptions for the same reason they include any other assumptions, because models where these assumptions aren't included don't do any better a job explaining any experimental results. If new experimental data shows these assumptions fail in some cases, then the models can be updated to handle them, same as any other failure has been handled in the past. (Granted each step becomes computationally more complex, so eventually we might hit a limit where it is too complex for a human to understand well enough to operate on.)
I think this is the best ways to talk about the difference between science as a process vs science as what people in white lab coats do.
People have a tendency to conclude that we should abandon science if something isn't quite right. It's better to think in terms of what will get us un-stuck from a local minimum/maximum.
This is an important point, but fortunately nature has provided us with a solution. Namely that teenagers are defiant and look for ways to distance themselves from their elders.
This is Planck's principle that science advances one funeral at a time [0]
> A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die and a new generation grows up that is familiar with it ...
> An important scientific innovation rarely makes its way by gradually winning over and converting its opponents: it rarely happens that Saul becomes Paul. What does happen is that its opponents gradually die out, and that the growing generation is familiarized with the ideas from the beginning: another instance of the fact that the future lies with the youth. — Max Planck, Scientific autobiography, 1950, p. 33, 97
The biggest impediments have historically been multi-generational organizations whose power requires limiting access to science. Famously the centralized church in the time of Galileo. More recently the cigarette and oil industries. Things like big personalities and legislative priorities tend to have much shorter time scale and usually allow for science to ratchet forward generationally. Big personalities die, legislators turn over every few years in the best case or in a few generations in the worst case.
Intra-generational power structures are a much larger impediment. Funerals take way too long to happen.
That is strange since to work in science today you have to be extremely compliant for close to a decade. Most people who aren't gets kicked out or removed in selection processes, what you have left is one of the most compliant parts of society.
Disclosure: Old person
Isn't that generally what convergence means? At least in the math sense, we are talking about t at infinity, meaning that in any real world time limited application, there is always a gap.
Everyone claims its different in their field
The journey to discovery is often just as fascinating as the results themselves. The process (the false starts, debates, even the dead ends) can be incredibly instructive and inspiring.
Also this is called research. You don't know the answer before head. You have limitations in tech and tools you use. You might miss something, didn't have access to more information that could change the outcome. That is why research is a process. Unfortunately common science books talks only about discoveries, results that are considered fact but usually don't do much about the history of how we got there. I would like to suggest a great book called "How experiments end"[1] and enjoy going into details on how scientific conscious is built for many experiments in different fields (mostly physics).
[1] https://press.uchicago.edu/ucp/books/book/chicago/H/bo596942...