BTW: I am not criticizing the actual reviewer of the paper. Data availability should be a policy of the journal and required for publication. How can you improve quality without access to the data? In this instance requiring data would have likely prevented the paper from even reaching peer review and wasting a busy individuals time.
Requiring (raw) data is relatively new, and is only in a few fields. For most of the history of peer review data was not required. In fact, none of the PIs I worked in during grad school would ever agree to releasing the raw data. I'm pretty sure they're still not in most of physics.
IMO, it's reasonable to require it, but understand that this is a relatively new requirement and not part of peer review culture.
That on its own is scandalous
I was reading of the "reproducibility crisis", in computing, in the nineties
Science still wants to be the warm friendly club, whilst playing competitively for high personal stakes.
Cannot be both. Fraud, deliberately cynical fraud and hopeful naive fraud, has been rampant for decades
The journals benefit directly and seem to be utterly complicit. The system is dreadfully broken
Nobody should wonder why people do not trust science
Science isn't a religion, nor is it a paper, thesis, or journal. It's a method for understanding the world around us through rigorous application of testing and retesting.
Science can be trusted. Measurements and the testing and retesting can be trusted. It's people that can't be trusted.
We call people in the field of Science, Scientists. Perhaps we should stop labeling fallible and untrustworthy humans as such. Science is trustworthy. Humans are not.
This is an important point and I think people should make it more often.
A human scientist who is committing fraud is misrepresenting facts about the world. And the way you would accurately describe those facts is with science. It's not that science is in the wrong here, it's that the human is lying about what the science says.
You want to be able to say "objectively, this guy lied about his research." But if you want to say that, then you need to be able to accurately describe both the world and the research so you can compute the diff between them. So if you throw out the science with the human scientist, then you've thrown out any objective means by which you were going to say the human scientist did anything wrong in the first place.
>Science can be trusted. Measurements and the testing and retesting can be trusted.
I still wouldn't go so far. Science doesn't provide a pure understanding of the world it produces at best a model based on observations and axioms. These both can be wrong without politics,money, or personal bias. Our senses are fallible and sometimes you measured the wrong thing with the wrong tool. You shouldn't trust science, you scrutinize it and use what you find works until it doesn't (and then publish that).
Taking an iterative approach where you make a prediction, test the prediction, and adjust course in response to the test (AKA science) is literally the only possible thing that can work to discover true things about the world. So if someone distrusts science they're basically giving up on a belief in objective reality.
But I agree it's not surprising that people don't trust science, any more than it's surprising that people believe in astrology, lizard people, or ESP. Human minds don't have a special science sensor that lights up when they read true things. It's pretty much garbage in garbage out.
Don't miss the story here, which is that the study failed to replicate and the fraud was detected. In many human endeavors there are frauds that have persisted for centuries.
> That on its own is scandalous
Yeah it is, I agree. Providing data should be mandatory to the reviewers. It should IMO also be provided publicly if feasible (at least upon request if it's financially burdensome to host) if the work is publicly funded.
I am not in the field, so I cannot comment on the specific relevance of the request, but in my area, there are absolutely some mandatory data inclusions which should block a paper from being published. Presumably this was not on that list.
Nature is a "premier journal" known for publishing cutting edge research. Any data - in this case a bunch of numbers - used to support the conclusion should be made available. I am sure there are disciplines where the evidence would be difficult to provide directly - fossils, physical items etc. Maybe this PI did the experiment himself - thought he won a Noble - only to find he could not in fact reproduce the results after publication - oops. IMO Nature did not do its job. Just pointing that out.
[1] https://plos.org/resource/how-to-store-and-manage-your-data/
Maybe it should be. "Peer reviewed" is taken as a sign of reliability in many contexts, including public policy. If it isn't actually such a sign, then either it needs to change so it is, or we need to stop taking it as a sign of something it isn't.
Even just reevaluating a submitted dataset is no guarantee. Someone committing an elaborate fraud could have faked the raw datasets as well.
What you're saying is, the first option I gave, to fix the peer review process so it is a good indicator of the reliability of research, is not feasible.
If that's the case, that just means we have to pick the second option: stop treating peer reviewed research as if it is reliable.
It's a system built upon honesty. A dishonest person is obviously going to take advantage of it. Really blatant things are caught in peer review, but sneaky fraud will definitely slip through.
Fraudulent research would obviously not be of the desired quality, wouldn't it?
> It's a system built upon honesty.
But if the purpose is to characterize the research in terms of quality, this can't be right. Since fraud is known to happen, and there are known to be incentives for fraud, any process that claims to judge the quality of research would have to have some way of checking for fraud.
Something can appear to be of very high quality and yet still be completely fraudulent. This occurs in countless fields; art, counterfeit money, deepfakes, etc.
Scientists are attempting to address fraudulent research in other ways, but peer review is not the best means for it and it should not be thought of in that way.
So in other words, peer review does not judge actual quality. It only judges apparent quality. Which is not what anyone actually wants to know.
It is the case, and that second option is exactly what we do. One article is nice, but replication is key before some new finding is accepted. That is exactly what happened with LK-99, for example, which is interesting because it was very public.
I don't think so. I see claims made all the time about science being "settled" purely on the basis of it being peer reviewed.
> replication is key before some new finding is accepted
If only that were true. In many cases it isn't.
Hence, it doesn't surprise me at all to see that he was the one to call out Dias. Some things never change!
We always joked that if you read it in Nature, it was certain to be wrong, but realistically, it meant that the topic was likely an important one with a small number of competitive groups who could all evaluate each other's papers and will loudly complain about every inconsistency or irreproducible detail.
Explain to me why Nature would publish a paper without having access to/requiring the data. Could it be Nature wanted to be the first to publish such a ground breaking discovery? Dishonest people are simply a fact of life. The entire point - supposedly - of peer review is to insure that scientist do not make unsubstantiated claims. Job well done Nature. Job well done.