Over the years, I've drifted to the biological side of bioinformatics methods. The papers I write are similar to CS papers, but the projects are bigger, and I no longer submit papers to CS conferences. My typical experience with peer review is accept after 1-3 rounds of revisions. Hence I get to see more of the positive side, where the reviewers act as editors and their suggestions improve the paper.
Of course, it's possible to get a CS-like experience with journals by being ambitious. If you try your luck by submitting the paper to prestigious journals it's unlikely to get in, your typical outcome is a desk rejection or rejection after a single round of reviews. Then you get to see more of the negative side of peer review. But I'm just not interested in playing that game.
Also hard agree on publishing reviews. It seems to be the only viable modification of peer review that we have.
My latest experience with peer review was just bad. No matter the conclusion they drew, there was just a clear lack of quality and understanding in their thoughts and effort. And this is not infrequent.
More generally, such a modification has knock on effects that probably warrant some thought on what we do with our publication practices. Published reviews will probably require more work from reviewers who are already time poor. Better reviews, hopefully, but fewer of them and fewer papers?
Counting a good review as a citable publication might be worth considering, as essentially mini idiosyncratic literature reviews. Perhaps in combination with Registered Reports[1], where the proposal is published and reviewed before the study/work is done, in which case the reviewer is closer to symbiotic co-publisher.
Either way, it seems that for both authors and reviewers, who are effectively the same people and both rely on peer-review as a guarantee of the value of their work, that the actual work of peer review needs to be taken more seriously and not conceived of as some sort of feudal aristocratic gentlemen’s duty.
~~~~~~~~
In some fields like AI knowing the exact authors is also quite common. Because the vast majority of researchers are employed by just a handful of big labs, the reviewers (and sometimes organizers) with experience in the subfield are employed by exactly the same lab(s). So they already know each other and each other's research anyway. Add to that that citations and prior work usually give away the set of authors too.
One thing I really wish is that we ensured that code works properly. I don't expect a full training but at least verify checkpoints.
What's the difference between posting stuff on arxiv or uploading a pdf to your blog?
> Big labs have big publicity and thousands will read their papers.
Peer review is supposed to circumvent the problem of a marketing budget being a factor in whether your paper is deemed relevant or not.
There isn't. My point is that the papers aren't blind because they've been read before. Publishing onto your blog would have the same result. I'll note that this isn't common practice in other research communities.
> Peer review is supposed to circumvent the problem of a marketing budget being a factor in whether your paper is deemed relevant or not.
I don't see how this could even happen. Let's even say you are a no name lab and get accepted, you're in a list of hundreds or even a thousand other papers. There's publicity, yes, but it isn't massive.
Let me put it this way. If you are a grad student at MIT or Stanford you will get hundreds of twitter followers relatively quickly. These followers are going to see you posting links to your papers. You have a larger network effect just by being at a larger and more prestigious school. That is the main advantage of prestigious schools and I don't think there's anything necessarily wrong with that.
But we do have to admit that there are factors that are highly important here beyond merit based. Being at MIT or Stanford does not mean your paper has more merit than a no name school or lab. That no name lab will not have the same reach, which results in fewer citations (our main metric of relevancy), and fewer researchers will be able to expand upon the work.
The major problem here is that we're pretending that this doesn't happen. I don't know how to solve this issue or even know if it can be solved. But it is really dumb to go around saying everything is merit based and not acknowledge the other factors at play here. It is pretending that there's a stronger signal in the noise than there is.
That misses the whole point of anonymous reviews, or even anonymous voting.
You want your work to stand by itself. Otherwise you're blasting open a very corrupt door where appeals to authority, petty politics, careerism, and funding play a role in how a paper is approved.
> Instead, in CS, there is a tendency to hide behind an abrasive negative review when the reality is that the reviewer does not understand the paper.
Isn't the whole point of a paper to present a topic in a clear and understandable way, so that your peers are able to cut through the bullshit?
If you pick a journal to publish your paper, which means you explicitly want the paper's editors to go through each and every single line of text you wrote to poke holes, but in the end once those holes are picked you complain that the journal you picked is not the right one for your flawless paper and that they all suck and their problem is that they don't understand your genius, what does this say about you and your work?
When you pick a journal you pick the subset of your peers to review your work. If your peers point out problems then why not listen to them?
Your comment reads a whole lot like "the fox and the grapes".
This door is already open, and impossible to close [1]. For any reviewer closely familiar with the area the paper is in it is trivial to guess which group the paper came out of 9 times out of 10. Style, approach, citations, and just knowing what the other people in your area are working on all contribute. To achieve any sort of blindness you have to go to reviewers who don't know that area as well... which, I suppose, does decrease the potential for corruption--at the cost of the reviewers not understanding the paper as well.
> Isn't the whole point of a paper to present a topic in a clear and understandable way, so that your peers are able to cut through the bullshit?
Even a well written paper often requires some effort to understand, due to conceptual novelties, mathematical complexities, etc.
Unfortunately there's little incentive for reviewers to put in much mental effort. Some of them will. Many won't.
> When you pick a journal you pick the subset of your peers to review your work. If your peers point out problems then why not listen to them?
Because the "problems" are often bullshit. You're assuming the reviewers are acting in good faith and finding legitimate problems with the paper, but they're often not--usually not out of malice, just laziness. There's no incentive to put any effort into a review above the minimum to convince the editor that your review is plausible. This doesn't take much since the editor is probably only going to glance briefly at the reviews. If you can finish your review in an hour instead of a day you can spend the rest of the day writing your own papers or doing other work that actually furthers your career.
[1] In all but the largest, most active areas, where there are so many people working that nobody can keep up.
For example, in fields where the peer-review is done primarily by academics are there better reviews during academic breaks vs. say, exam season? Does this translate to more accepted work because the reviewer can spend more time digesting the content?
Second, journals are overloaded, the journal reviewers unpaid, and frankly, they are not doing a good job anymore in reviewing work. Feedback that amounts to "maybe submit first to a conference" is not that valuable. Effectively, you have to have the idea, then execute it perfectly, then do a major marketing effort to popularise your work, and why exactly would you at the end of this still submit to a journal? Given that the journal either wants the rights to your work, or charges you for the pleasure of publishing with them. At this point, a journal is just another marketing step, and for marketing you pay, I guess.
Yes, but there is unfortunately a wide variability in the knowledge of reviewers. It's not uncommon to have reviewers directly contradict each other about the shortcomings of the paper. E.g.,
Reviewer 1: "Section A needs to be condensed, this level of background is unnecessary. Readers will find this as common knowledge."
Reviewer 2: "Section A should be expanded as all readers may not be knowledgeable in this area."
Particularly for grad students, this can make the process feel arbitrary and the feedback of limited value. There also can be an inordinate amount of stylistic review comments (e.g., "however" should be changed to "nonetheless"). These often come across as the reviewer feels the need to critique something because they don't have sustentative critiques. Good critiques strengthen a paper, bad feedback seems to do nothing but check the process boxes.
I actually agree with the OP that one potential solution is to encourage reviewers to decline reviewing papers that they don't have the requisite knowledge to provide good feedback. Unfortunately, this decision is often only offered based on reading the title and abstract.
The meta-reviewer doesn't just send the reviews to the authors with no commentary. They write a summary of the reviews with a recommendation about what to do next. If there are contradictions and no consensus is reached, it is their job to decide which concerns the authors need to respond to in order to get it published. They aren't bound by the reviews, they can disregard one if they feel like it is unfair, although sometimes the rules say they must get another reviewer to write a 4th review.
The open discussions not only benefit the peer review process but also act as context and learning material for others.
Never heard this argument, but having been under many double blind peer reviews in the last 10 years, I can sign this.
Interestingly, in my last paper for PLOS One, one of two reviewers chose to lift anonymity and specifically his comments were on-focus, supportive and substantial.
As a consumer of papers now, I find peer review pretty useful. I can’t read everything that comes out. Even if I just read abstracts, I need some sort of filter. Publication in a top tier journal raises the probability that egregious errors haven’t been made and that the paper is worthy of my time.
Of course, in practice, this has the problem that it is already hard to get reviewers, because nobody has time anymore. Removing the anonymity of reviewers leads to the judge becoming open to judgement themselves, and many would not like that. Furthermore, there might be severe repercussions if the reviewed is more powerful than the reviewer, especially in authoritarian societies. But maybe one should just exclude these societies from peer review, and let them do their own thing.
I think peer review has to become a market place. Let everyone choose for themselves which paper they would like to review, and there is both positive and negative credit for both reviewers and authors. The pool of reviewers of a paper shares 25% of the credit that the authors of the paper get. This way you are incentivised to review even outlier papers, because if they are successful you get a substantial amount of credit for them. How the credit is distributed among the reviewers should also incorporate a time factor, so that reviewers rushing to a paper that is already successful don't get nearly as much credit as a lone early reviewer.
I suppose there can be some other layer of reviewer meta-review to account for this, much like the role 'acceptance rate' has come to have for journals and conferences. But, following Goodhart's law, even that has come to be gamed now that a meaning - a proxy for journal prestige or quality - has been placed on it.
I quite like the fact that whatever someone says on the internet now, the first comment is usually the best-argued counterpoint. It seems like a dynamic that could be brought to science quite well. I may not have the time to parse an entire paper in a field unfamiliar to me, but if I come across it and the top comment is 'there's a math error in table 6 that throws paragraph 4 into doubt', I can certainly verify that for myself. That sort of 'peer' review would make science much more accessible to the interested layperson.
Hence, they will sometimes try to make sure you don't publish or at least get downgraded to sister journals, such as Nature Communications.
Another problem are editorial decisions. Great work is often desk rejected because it's judged not to be significant or interersting enough (usually the author comes from a less prominent university).
For example, the original CRISPR article was desk rejected by most journals and the fact that the author's affiliation was a no-name research centre in Alicante (Spain) surely played a role. This delayed CRISPR getting widely known by a few years...
This is interesting. Has there been a write up on this saga anywhere?
I'm so glad to be out of that world, to be honest.
This problem goes even deeper, because it not only applies to papers but also to research grant applications. I even had once got a (positive) recommendation for one of my applications, but I could see from the statement that the reviewer did not really understand what was important.
That's not to say the proposal isn't worthwhile, just that it didn't find the right audience. A contractor may have a whiz-bang new method of snow removal, but it doesn't mean people in Phoenix will want it (or even understand it).
.) an unsolvable problem inherent to all peer reviews?
.) a problem that comes from specific practices, and could be solved by doing peer reviews differently?
I'm just a layman - but if reviews are not done by peers (= experts working in the same field)... who else could properly review a paper?
They are actually proposing that peer reviews are credited:
> Instead, authored and credited reviews might lead to more responsible reviews, or reviewers respectfully declining when they might not know the topic.
And yes, I do think reviewers should be credited. Because it's work, for one, that needed to be done so the paper could be published, and because it would probably indeed incentivize reviewers to do the work correctly.
However, it could incentivize them to reject more papers, in fear of accepting a bad paper and be publicly advertised as a reviewer who reviewed and accepted it.
This is leading to quite irresponsible reviews. Instead, authored and credited reviews might lead to more responsible reviews, or reviewers respectfully declining when they might not know the topic.
Instead, in CS, there is a tendency to hide behind an abrasive negative review when the reality is that the reviewer does not understand the paper. Programme Committees are relieved to find a negative review, however unfair or off-kilter it is, because more rejected papers will decrease the acceptance ratio of the conference, hence make it appear more competitive.
Double-blind reviews are just peer-review theater. It is quite simple to guess which group the paper is from. It is difficult to guess the exact set of authors, but reviewers who are out to settle a score or to discard dismissively just need to know a ballpark of where the paper is from in order to stonewall with an irascible review.