News for hackers. Stuff that matters.
Re:Why isn't preprint review being adopted? by PaulHoule on Monday March 25, 2024 @10:36AM (#39816871)
And also has limited value as we practice it today.

A useful review would involve:

(a) "This paper won't be accepted by Cochrane for meta analysis", "N=20 get out of here", ...

(b) Researchers provided their data files and Jupyter notebooks, the reviewer got them to run

(c) Reviewers attempt their own analysis for at least some of the data (think of the model of accounting where auditors look at a sample of the books)

(d) Reviewers come visit the lab and take a look at the apparatus

(e) Something like a PhD defense

(f) Summarize/formalize discussion of a paper that happens at a conference or online venue

(g) 5 months, 5 years, or 40 years later (once in my case) somebody goes through the math line-by-line and finds a mistake on equation 178 that propagates and affects the rest of the calculation. This knowledge ought to be captured and "stapled" to the paper.

I wouldn't say peer review is useless, I think it did improve papers I wrote a little but reviewers do not put enough effort into to reliably catch validity problems. If you believe in meta-analysis, which you should, read this book

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1114087/

the first one (a) is really important because the overwhelming majority of papers are excluded in Cochrane reviews for most medical topics. If 80-90% of papers in some fields are not suitable for inference that raises tough questions such as "Why does work like this get published?" and "Why does this work get funded?" If we got half as many papers but 80-90% of them were usable that would be a revolution.

Re:Why isn't preprint review being adopted? by rhelz on Monday March 25, 2024 @06:41PM (#39822060)
> If you believe in meta-analysis, which you should,

I view meta-analysis as being like those mortgage-backed securities which crashed the world back in 2008. I mean, yeah, theoretically, a bond whose yield is a weighted average of mortgages should be less risky than any one of the mortgages in it.

But....the devil is in the details. When I start seeing meta-analysis which claim that, e.g., masks don't protect you from a respiratory illness which spreads by coughing and sneezing, or that vaccines are at best worthless and at worse cause autism, wellll.....

....I have to conclude that garbage-in, garbage-out.

Re:Why isn't preprint review being adopted? by rhelz on Monday March 25, 2024 @06:55PM (#39822169)
> I wouldn't say peer review is useless, I think it did improve papers I wrote a little

Let's get back to basics here. What is the whole point of doing things scientifically in the first place? I.e. what is so special about the combination of 1) communicable descriptions of experiments, which can be re-run by anybody, and 2) the performing of said experiments and reporting the results, and 3) having others review and reproduce the results?

Why are those 3 practices soooo special? Its because they are designed to either a) persuade everybody to agreement with you that your results are valid, or b) persuade you that you are wrong.

I.e.. the scientific method is uniquely designed for the purpose of compelling agreement among peers. Its the 2nd best method we have of reaching consensus (the first best being the methods of proof used in mathematics. )

This is important because even if you make a new gee-whiz discovery, its useless unless other people agree that it is real. And unlike politics, or a beauty contest, or an aristocracy, it doesn't matter who you are, if your results are observable and reproducible, other scientists will agree with you, and your discovery has some chance of being generally useful. And that's why they are anonymous--its supposed to cut through the biases and prejudices which, alas, sometimes prevent us from coming to consensus.

The point of peer review is to get a read on whether or not the author of the paper has justified their claims in enough detail that the results are likely to compile agreement. Yeah, we see if 3 or 4 people can agree on it, before asking everybody to agree on it. Whoda thunk?

Point being, peer review isn't to help you improve your paper.

> reviewers do not put enough effort into to reliably catch validity problems.

Yes. This. Any why don't they? Because there's only so much work you can expect somebody to do for free. Lets pay them.