I spend most of my career reanalyzing others' data and combining it together into larger datasets, in molecular biology and genomics. The only time I encountered mistakes it was from my own labeling errors or bugs, pre-publication. And in pre-publications datasets I would sometimes detect accidental swaps on the labels of samples as part of QC checks.
“Sokal Squared” is a great example of the problems in the field.
https://en.wikipedia.org/wiki/Grievance_studies_affair
Regarding the reproducibility crisis:
As the saying goes, science advances one funeral at a time, but it does advance.
It seems overall you can still have an optimistic point of view of science while accepting it does not work as well as it could.
It's a race to the bottom, and how can we ask that people compromise the quality of their work to compete when the only motivation for doing this work is the love of the craft. It's certainly not the money, or the environment, or the work-life balance, or the prestige.
When in the trenches of science this is literally your job. What seems like the likely hypothesis, what model that explains the data is good or bad.
Papers are not textbooks. They are the boiling cauldron from with some bits eventually emerged as fully cooked facts.
The only way that I can square others' experience of science with my own is if they take the Discussion sections, which are free-form extrapolation about future directions, and treat them as if they were asserted as truth.
I'm talking about malicious, fraudulent results. Made up numbers, code obfuscated and manipulated to do things different from what's being claimed. I resent the implication that my issues stem from a naive, idealistic reading of the discussion section rather than a thorough examination of the methods.
Sorting the good from the bad is one thing, creating and evaluating models, good models or bad models is one thing. Having to compete AND CITE fraudulent work, lies, imaginary models to be able to participate in the system is not "literally" science. And it's not the job any self-respecting scientist should want to do.
The more money is involved, the more fraud to expect. This should not surprise: fraud goes where the money is. And where there is money, the stakes are higher.
But I have seen reprehensible behavior even in opponents of the Younger Dryas Impact Hypothesis, where practically no money seemed to be at stake.
To that end, does anyone know of any list of techniques for spotting or identifying fraud in scientific works? This is a subject that I wish somebody would write a book on, so I am curious as to what techniques people use for this purpose.
But: science isn't really self correcting. Someone has to step up and fight to correct it, every single time. Spend some time learning about this problem - as I have - and you will find the overriding attitude is one of despair. Science isn't self correcting, it's arguably self corrupting:
1. There's no incentive to pick fights with colleagues so it almost never happens
2. Even when someone does pick that fight the institutions go to ground and defend their people, so there's no outcome.
3. The first response of journals and universities on being informed of fraud is to turn around and tell the fraudster everything, so there's no way to keep fraud detection techniques secret.
4. And usually after that they ask the fraudster to submit a "correction" (i.e. higher quality fake). The idea that you should maybe NOT let a fraudster have a second try once caught, does not seem to occur to the brightest of sparks that run our scientific institutions. There are actually cases where this has happened and then the original complainants spotted that the newly submitted correction was also fraudulent.
How do these papers get spotted? Spend time on PubPeer. It's a site where people compare notes on dodgy papers and how they're being detected. Also read the blogs of people like Elizabeth Bik, Smut Clyde, read the old blog posts by Joe Hilgard when he was still in academia, follow https://twitter.com/steamtraen
tl;dr There's a range of techniques used for different fields, often by looking for re-used images or data across papers that shouldn't be re-using them, or by spotting internal inconsistencies in reported data tables.
Do all of them do this?
There appear to be no limits to how absurd this phenomenon can get. Here's an example:
https://pubpeer.com/publications/7611C2EEE5789028895D8BD02A1...
But after becoming a scientist, coming across so many fraudulent papers, so much non-reproducible and poorly done research. And seeing how it's affected my own, the thing that's dearest to me, having to build on top of those results and work in that global environment. It's been heart breaking, I feel no love for science anymore.