Arxiv.org reaches a milestone and a reckoning

Re:Arxiv.org reaches a milestone and a reckoning by UncleMeat on Tuesday January 18, 2022 @08:58AM (#29979142)

I think the core problem here is the layperson perception of peer review. People often say "it is a peer-reviewed paper!" when trying to claim that a paper is definitive and powerful. But when you speak to scientists they have a much more muted understanding of peer review. And I think that is okay. The amount of effort it would take to do a very thorough analysis of a paper is large, sometimes even approaching the cost to do the research in the first place.

In fact, peer review is often more about novelty and importance rather than rigor. Very well structured research will get rejected if it isn't seen as contributing to the field in a nontrivial way.

A few fields (psych is the big one) are funding replication grants. That'd be the true mark of quality.

[ Parent | on HN ]

Re:Arxiv.org reaches a milestone and a reckoning by marginalia_nu on Tuesday January 18, 2022 @10:00AM (#29980129)

> In fact, peer review is often more about novelty and importance rather than rigor. Very well structured research will get rejected if it isn't seen as contributing to the field in a nontrivial way.

This is actually deeply problematic, as a part of what's driving the replication problems is specifically publication bias. If the only thing that gets published are unexpected or noteworthy results, you're selecting for statistical aberrations.

[ Parent | on HN ]

Re:Arxiv.org reaches a milestone and a reckoning by UncleMeat on Tuesday January 18, 2022 @11:28AM (#29981533)

I don't really agree. I think it is valuable to push novel and influential results into visible venues and to reward work of this kind. It is critical to understand how this introduces bias into the process and to consider this when reading a given paper, but I'm not sure that a review system that only considers experimental rigor would be desirable.

[ Parent | on HN ]