Reproducibility trial: 246 biologists get different results from same data sets

Re:Reproducibility trial: 246 biologists get different resul... by mcphage on Friday October 13, 2023 @08:18AM (#37869769)

This doesn't seem like quite the same issue as the reproducibility problems in psychology, for instance. Researchers were presented with some data, and they came up with different analyses for it. That doesn't seem especially surprising—for a complex phenomenon, where you don't have a ton of data, and maybe the thing you're studying isn't very strong—yeah, you're going to get different results.

It's like people arguing over which sports team is better—the facts of the game are the facts of the game, but there's plenty of room for disagreement over what stats are most important, who has an easier season—and even if the two teams were to play each other directly, it's not always that the "better" team wins.

But the reproducibility crisis in social sciences seemed much deeper—one team would say that some effect exists, and other teams would try to reproduce the results and come up with nothing.

[ Parent | on HN ]

Re:Reproducibility trial: 246 biologists get different resul... by tgv on Friday October 13, 2023 @08:42AM (#37869935)

There is a relation between them, but not a strong one, I agree. The more conclusions you can draw from a data set, the more will be wrong. A possible reason, hinted at in the article, is underestimating the noise. It's one of the things that went bad in psych studies, but also in genetics (in a lecture some 10 years ago, a statistician explained how results in genetics would be reproducible only at 5 sigma or higher).

[ Parent | on HN ]