The most important thing is that everything about the investigation is open: the methodology (including everything not found in the paper), the programs, the data. It's more like posting code for a large project on Github and then new researchers make PRs to correct bugs or extend the program in useful ways.
Yes, I expect journals to have to have correct results - not in the sense that the generalizations from their findings are universally valid, but in that the statements of fact and of logical implications are valid.
To be more explicit:
* "Events X, Y, Z occurred" <- expect this kind of sentences to be correct
* "We did X, Y, Z" <- expect this kind of sentences to be correct
* "A and A->B, so B" <- expect this kind of sentences to be correct
* "We therefore conclude that X" <- Don't expect X to necessarily be correct.
See for example https://slatestarcodex.com/2019/05/07/5-httlpr-a-pointed-rev... for a slightly different take.
This is troubling to me. It does not seem like the bias was due to something arcane, some finer point in advanced statistics, something hidden from view etc. The poster says:
> It involved regression towards the mean — when noisy data are measured repeatedly, values that at first look extreme become less so.
This may not be 100% straightforward when it's buried in the middle of a paper, but if you actually consider the methodolgy you are likely to notice this happening.
So here's what _I_ learn from this case:
* It is possible that the reviewers at Nature don't properly scrutinize the methodological soundness of some submissions (I say "possible" since this is a single example, not a pattern)
* PhD avisors, like the author's, may not be exercising due dilligence on statistical research done with their PhD candidates. The author's advisor had this to say:
> "It's great that we’ve persisted in attempting to understand our methodology and findings!”
so he says it's "great" that they did not fully understand their methodology before submitting a paper using it. Maybe that's not exactly what he meant, but still, pretty worrying.