Re:Is Peer Review a Good Idea? (2020)
by humanistbot on Wednesday September 14, 2022 @05:51PM
(#32843901)
In some publications (mostly CS conferences), there are associate chairs or 'meta-reviewers' whose sole job is to manage the peer review for a paper. They will read the reviews and if there isn't consensus, they'll ask the reviewers to read the other reviews and have a discussion. It is usually 3 reviewers and 1 meta-reviewer per paper, so there are often 2-1 splits.
The meta-reviewer doesn't just send the reviews to the authors with no commentary. They write a summary of the reviews with a recommendation about what to do next. If there are contradictions and no consensus is reached, it is their job to decide which concerns the authors need to respond to in order to get it published. They aren't bound by the reviews, they can disregard one if they feel like it is unfair, although sometimes the rules say they must get another reviewer to write a 4th review.
Yes, but there is unfortunately a wide variability in the knowledge of reviewers. It's not uncommon to have reviewers directly contradict each other about the shortcomings of the paper. E.g.,
Reviewer 1: "Section A needs to be condensed, this level of background is unnecessary. Readers will find this as common knowledge."
Reviewer 2: "Section A should be expanded as all readers may not be knowledgeable in this area."
Particularly for grad students, this can make the process feel arbitrary and the feedback of limited value. There also can be an inordinate amount of stylistic review comments (e.g., "however" should be changed to "nonetheless"). These often come across as the reviewer feels the need to critique something because they don't have sustentative critiques. Good critiques strengthen a paper, bad feedback seems to do nothing but check the process boxes.
I actually agree with the OP that one potential solution is to encourage reviewers to decline reviewing papers that they don't have the requisite knowledge to provide good feedback. Unfortunately, this decision is often only offered based on reading the title and abstract.