Not to mention the corollaries from the Wikipedia article (https://en.wikipedia.org/wiki/Why_Most_Published_Research_Fi...):
"In addition to the main result, Ioannidis lists six corollaries for factors that can influence the reliability of published research.
Research findings in a scientific field are less likely to be true,
1. the smaller the studies conducted.
2. the smaller the effect sizes.
3. the greater the number and the lesser the selection of tested relationships.
4. the greater the flexibility in designs, definitions, outcomes, and analytical modes.
5. the greater the financial and other interests and prejudices.
6. the hotter the scientific field (with more scientific teams involved)."
These are all reasonable criticisms from my own area of expertise: applied econometrics.
I won't speak for all textbooks, but generally stuff you find in there should not be the same as what you find in journals, and is much more settled. Big caveat that that isn't necessarily true for younger sciences without long-established theory, say exercise physiology or social psychology, but something like a chemistry textbook is pretty damn trustworthy.
And those are what people who aren't actually scientists should mostly be educating themselves with, not newspaper science reporting sections.
Academia is a complex place and it's full of fake results to get publications. (I'm coauthor of several scientific papers and I know for a fact countless highly rated papers in my former field are not reproducible because they come from adjusted numbers).
Individual papers where never intended to be the final arbiters of truth, that’s not their role. If nobody thinks things are worth looking into again then stuff stays in a very nebulous state which is no worse than where things where before a paper was published.
I don't think so, because of the aforementioned idiots who take a study that can't even be replicated but then cite the paper as evidence for something. If it can't be replicated, logic would say there is a very good chance it's inaccurate. Yet it's treated as being "settled science" which is worse.
Also, many times it isn't that nobody thinks it's worth looking into again, it's that coming to "the wrong conclusion" on some things will end their careers so they'd rather play it safe and just keep sucking up grant money to pump out more of the same garbage.
Mistakes can be made, data can be limited or misinterpreted, scientists can be corrupted. Theories, "common sense" can change: what we thought we'd know for sure have been proven wrong. If you are a scientist, you know that.
Only because I don't automatically take every "scientific" finding at face value, it doesn't mean I don't trust science.
I trust science, I just don't trust every single research, experiment, scientist, or journal. Actually that's in itself, in a way, science.
There is plenty of things that we took as scientific fact in the past that turned out to be false. Science is supposed to question existing notions and test/prove new theories. If you just assume that we're in a post-science era where we've got it all figured out, and our current theories are all correct, that's not science, it's dogmatic faith.
Ignaz Semmelweis was destroyed by the medical community for his insane idea that doctors should wash their hands before performing medical procedures. He was attacked about to the point where he ended up having a nervous breakdown and was committed to an asylum where he was beaten to death by the guards. Serves him right for questioning the science!
The process by which DNA was convincingly demonstrated to be the molecule of hereditary was fairly complex; an early experiment that was complex and hard to understand did so, but people didn't completely believe it so a later experiment that was easier to understand was done.
For the longest time, the establishment believed the functionality of the ribosome (a critical subsystem that translates mRNA into protein) was carried out by its protein subunits. Although convincing data was published in the 1960s, the general belief did not change until the crystal structure of the ribosome was published showing that RNA formed the catalytic component.
And my personal favorite, it was considered unpossible that prions could be caused by proteins that misfolded and caused other proteins to misfold, it required absolutely heroic efforts in the face of extraordinary pressure to establish the molecular etiology of prions in the minds of the establishment.
It's hard to change your mind. Some people never will.
The statistics underlying all of this make a number of implicit assumptions that may or may not be true IRL (for example: that journals publish papers independent of the salaciousness of claims made within them). If Science picks out only the top-5% most-sensational claims for publication, then you can't assume that a 95% CI is a safe threshold. You've probably got to increase it to a much higher value to have any prayer of getting past the inherent bias in such a process.
When nearly every policy can be backed by "some papers published in <some journal>", stuff like this becomes important. For a recent example, the US White house is engaging in a 5 year study to determine if releasing sulfur dioxide into the stratosphere would accomplish enough "cooling" to offset the "climate crisis". the "climate crisis" is something that exists in computer models, few of the models agree, and if they do, it behooves us to look at when they started agreeing and find out why that change happened.
a more personal example: I can't reproduce getting CaCO out of seawater, yet.
Second, sociology and anthropology are among the least empirical of academic disciplines. What reason would you have to involve them?
Third, many studies are of the type "my theory predicts X and my experiment shows p < 0.05". That's almost wrong by definition. Bayesian theory explains why.
Fourth, many studies have conclusions/claims that cannot be inferred from the actual finding.
> When nearly every policy can be backed by "some papers published in <some journal>", stuff like this becomes important.
It is important, but as long as academia is a "turn out papers for tenure" industry, it won't be fixed. Don't take the conclusions of a paper for true. They're meant for other academics.
And i think that R and python being popular in the academic fields belies the fact that no one wants to talk to statisticians about their data. It's a lot easier to massage inputs to a machine than a human.
I was under the impression we were talking about reproducibility in this thread; therefore my point about policy being influenced by something you suggest is "meant for other academics"
With math, physics, applied math, computer science, etc, bad results can be usually (but not always) deconstructed by anyone with the merest tools of the rational; formal logic, application of various categorical laws, etc.
Science as a process is useful, but its conclusions can only be as reliable as what its body of producers are incentivized to produce.
There's a reason you don't see a crisis in e.g. physics, where their hypotheses are leaps and bounds more testable.