News for hackers. Stuff that matters.
Re:Understanding Bias in Peer Review by epistasis on Thursday November 30, 2017 @05:32PM (#15820233)
I'm extremely glad that they measured this, and are sharing it. Even though I think it's obvious ahead of time, some people will be surprised about it. Which makes it doubly important to share, IMHO.

Why does Google get a ton of press for so many things? Because they are Google and they attract attention. Why does MIT get so much press for so many things? Because they are MIT and they get a ton of attention. Reputation matters a lot and is self reinforcing.

Peer review is only partly about saying "this seems to be well evaluated science." For journals and conferences, it's much more about "this is interesting to the audience." That is a hugely subjective evaluation, based on all sorts of cultural clues.

It would be extremely odd if scientists were somehow able to ignore that human tendency in peer review. If you can see names and institutions, and you have reputations associated with them, that will without doubt enter into the highly subjective evaluation of peer review. I would guess that even if you know about this human bias in yourself, you would still be subject to it subconsciously. I'm not sure there's any way to get around it except things like double blinding. Even then, for many sciences, the techniques and data sets used may very well give away who did the work.

Re:Understanding Bias in Peer Review by benbreen on Thursday November 30, 2017 @06:38PM (#15820681)
I’m in history but it’s the same problem in our field - all the peer reviewing I’ve done has been double blind, but I can guess the author perhaps half of the time based on things like self-citation. It’s a hard problem to solve, for sure.
Re:Understanding Bias in Peer Review by Vinnl on Friday December 01, 2017 @06:43AM (#15823273)
Yeah, double blinding is ineffective in almost every field. A good article on double blind peer review is [1].

Being aware of the bias is one thing, although I'm afraid that indeed only partially mitigates it. Making peer reviews public after the fact could help somewhat in allowing people to judge for themselves and make the biases less damning, likewise for post-publication peer review. In any case, that would make it more feasible to do research on peer review and perhaps find better solutions.

[1] http://blogs.plos.org/absolutely-maybe/2017/10/31/the-fractu...