News for hackers. Stuff that matters.
Re:The Arsenic DNA paper exposes flaws in peer review by rubidium on Thursday October 03, 2013 @03:55PM (#6491556)
``we need to get past the antiquated idea that the singular act of publication – or publication in a particular journal – should signal for all eternity that a paper is valid, let alone important''

Most scientists I know (physics) don't have this idea. They know publication is just a step. Most grad students learn at some point that just because it's published doesn't mean it's correct.

Publication is a checkpoint, not a finish line, for any theory or experimental result.

Re:The Arsenic DNA paper exposes flaws in peer review by jofer on Thursday October 03, 2013 @04:23PM (#6491747)
Very true, but he does go on to state:

> If we had, instead, a system where the review process was transparent and persisted for the useful life of a work

That is a very interesting idea. (Reviews are currently confidential and usually anonymous -- only the author and editor sees the reviews, and only the editor knows who wrote them.)

Not only would this reduce blatantly petty reviews, it would also give some recognition to the work that goes into reviewing a paper. It's currently an important but very thankless job. It's considered polite to acknowledge reviewers at the end of the paper (even if they remain anonymous), but it's often not done.

Re:The Arsenic DNA paper exposes flaws in peer review by arjunnarayan on Thursday October 03, 2013 @04:37PM (#6491846)
I'm going to respond to this in the context of academic computer science (which works off conferences), in particular, the systems subfield. YMMV if you extrapolate this to non-CS science and journals, so take this with a grain of salt.

The bottom line is I, as a powerless grad student, am _very_ comfortable with the level of reviewing competence I have witnessed (and been a part of).

The proposal to deanonymize reviews has a huge flaw: Namely, there's no way I (as a grad student with no political power) will write a critical review of somebody famous if I know I won't remain anonymous.

The risk of blowback is too damn high. (And given the small pool, writing style would probably be sufficient to deanonymize, if my entire trail of reviews were public). There's no way I have the balls to write a critical review of a paper by, say, Robert Morris or Larry Peterson, for some prestigious venue such as SOSP, when I'm asked to write a review. Unless I know for sure that there's absolutely no risk whatsoever of being identified.

Look. Reviewing is thankless and hard. Anonymity exists so that we can be critical and not risk career suicide. Last year at OSDI I received 9 reviews, and they were all exemplarily thoughtful, even the obligatory "strong reject" one of them. The problem of "petty reviews" _has_ a solution: The program committee is self policing, and the reviews are not anonymous within the PC. (For example, at SOSP this year, there are 28 PC members.) You know that most of your reviews come from this gang of 28. You might have one or two externally solicited experts in addition, but that too is not anonymous _to the PC members_. If there's a culture of meanness, the PC members and PC chairs can fix this by internal policing (and they do).

Re:The Arsenic DNA paper exposes flaws in peer review by jkimmel on Thursday October 03, 2013 @06:47PM (#6492685)
Your point regarding anonymity is highly valid, and I appreciate that you bring it up. There is actually a concept I'm rather fond of that seeks to address this, usually referred to as peer-reviewed peer review [1].

Essentially, anonymous experts review publications in much the manner they do now, with no risk of their identities being revealed. A separate set of review-reviewers then rates the publication-reviewers themselves, scoring them on a standardized quantitative scale (1-10) based on the validity of their claims and the depth of their assessment. In this manner, experts collect an "aggregate reviewer rating" of sorts analogous to an h-index [2], call it an R-index.

This R-index can then serve to select accurate reviewers for future publications. A high R-index could also be included as a positive bullet point on a CV, providing some reward for reviewers to put a high degree of effort into their otherwise unrewarded reviewing work.

A few potential flaws are evident: what is one to do if early reviews are scored poorly, and you are no longer able to provide new reviews to increase your R-index? A grace period before scoring could potentially remedy this, ie) an R-index is not assigned until a reviewer has written >10 reviews.

[1] - http://www.frontiersin.org/Computational_Neuroscience/10.338... [2] - http://en.wikipedia.org/wiki/H-index‎

Re:The Arsenic DNA paper exposes flaws in peer review by arjunnarayan on Thursday October 03, 2013 @06:56PM (#6492719)
I like the R-index idea. It's basically Slashdot's meta-moderation system? I always thought that had some promise. However, it does add even more _work_ to a system that is extremely stressed (writing reviews is so much work! Imagine having to read papers, read reviews and then review the review on top of that!)

But basically existing program committees (in computer science) do serve that meta-moderation purpose (without the empirical R-index metric). The entire PC does get to see the deanonymized reviews of the other program committee members. If someone's just being an unhelpful jerk, they're probably not going to get invited back in the future.

Re:The Arsenic DNA paper exposes flaws in peer review by jofer on Thursday October 03, 2013 @04:48PM (#6491926)
Very good points!

I completely agree that there's a lot value in anonymity when reviewing, and I don't think it should go away. Blowback over disagreeing is very, very real, and I've experienced it myself (though it was from presenting something at a conference that disagreed with part of a previous publication, rather than as a result of a review).

However, I do think public and optionally non-anonymous reviews could be helpful.

(Incidentally, you had _9_ reviewers for a single publication?? Holy cow... In the geosciences, I've never heard of more than 3, and it's usually only 2. That's one thing that would really help in my field. More reviewers results in a broader review, rather than a particular person's pet peeve.)

Re:The Arsenic DNA paper exposes flaws in peer review by arjunnarayan on Thursday October 03, 2013 @05:27PM (#6492203)
To be fair, it's not common to have 9 reviews. OSDI 2012 was particularly thorough (my reviews totaled about ~4000 words) I'm more used to 4-6 reviews, around 3000 words or so. SOSP/OSDI is probably the most thorough and competently run conference that I know of in all of academia. This makes sense: they only take about 30 papers (it's restricted not just by quality, but the quality bar goes up as more submissions are made, just to keep it exclusive), it's once a year, and systems researchers sync their lives to the OSDI/SOSP submission cycle. It's the "flagship" systems conference, and a lot of advances are first described as OSDI/SOSP papers. For some perspective, the original MapReduce paper was by Jeff Dean and Sanjay Ghemawat, published in OSDI 2004. Last year's best paper was the description of the Spanner Distributed Database by Google.

Incidentally, I have deanonymized several reviews already. When you go to conferences, and some researcher asks the exact same question and has the exact same opinion as reviewer C, chances are... Sometimes the recognition is mutual as they realize what they just revealed (thankfully that has only resulted in humorous chuckles, at least in my life. But I can see how it can go wrong so easily!)

Re:The Arsenic DNA paper exposes flaws in peer review by seanmcdirmid on Friday October 04, 2013 @12:28AM (#6493709)
You might only get 9 reviews though if you pass into the second or third rounds. Controversial papers in the PL conferences I haunt will get more reviews out of necessity; better to add more reviewers to help break ties.
Re:The Arsenic DNA paper exposes flaws in peer review by arjunnarayan on Friday October 04, 2013 @11:21AM (#6495848)
Yes. According to the OSDI 2012 proceedings introduction, all papers in the third round had 9 reviews. No paper was accepted or rejected with less than 51% of the PC member's assent.
Re:The Arsenic DNA paper exposes flaws in peer review by seanmcdirmid on Friday October 04, 2013 @01:20PM (#6496642)
That is good and bad, it definitely skews toward safe non controversial papers.
Re:The Arsenic DNA paper exposes flaws in peer review by scott_s on Thursday October 03, 2013 @06:00PM (#6492428)
Interesting. I don't think I do, or would have, feared blowback from giving a negative review - but I have no way of knowing for sure. I do know that I'm generally reviewing papers because a colleague asked me to, or asked me to be on the PC. So, in that instance, I'm more concerned with doing a good job in the eyes of my colleague.

I do know that I have received many cursory reviews where the reviewer did not fully read the paper. I have often wondered if single-blind the other way would be better: reviewer signs their name, but paper authors are anonymous. While this may fix the crappy review problem, it is susceptible to what you fear.

Re:The Arsenic DNA paper exposes flaws in peer review by arjunnarayan on Thursday October 03, 2013 @06:10PM (#6492481)
Some reviewers do (voluntarily) do that[1]. I would seriously consider doing so once I'm tenured (i.e. when I feel I am immune to blowback).

I would hope that those cursory reviewers resulted in feedback from the other PC members that fix the issue (either through improved future reviews, or not being invited back to be on the PC.)

[1]: http://se.ethz.ch/~meyer/publications/online/whysign/

Re:The Arsenic DNA paper exposes flaws in peer review by scott_s on Thursday October 03, 2013 @07:03PM (#6492740)
I've heard of others who do that, too. The difficulty is that those who self-select to sign their own name are probably not part of the problem.
Re:The Arsenic DNA paper exposes flaws in peer review by seanmcdirmid on Friday October 04, 2013 @12:31AM (#6493718)
I sign my reviews unless I'm involved in some sort of post paper evaluation. It seems only fair since the reviews aren't double blind (given double blind, I would not sign). I also publish reviews on my own papers as a matter of interesting record.
Re:The Arsenic DNA paper exposes flaws in peer review by drjesusphd on Friday October 04, 2013 @08:53AM (#6495022)
I agree. But why not make it double-blind? Make the author anonymous to the reviewer too? Wouldn't that make the most sense?
Re:The Arsenic DNA paper exposes flaws in peer review by gizmo686 on Thursday October 03, 2013 @07:02PM (#6492736)
>only the editor knows who wrote [the reviews].

This is not entirely true. I've talked with several people involved in research (math and CS). The ideal is that only the editor knows who wrote the reviews. However many fields do not have that many people who are qualified to review cutting edge results. This means that it is possible to get an idea of who your reviewer is based on how he wrote. Of course, although the people I talked to said they often can figure out who reviewed them, they cann't actually confirm that they are right (and I definitely cannot).

Re:The Arsenic DNA paper exposes flaws in peer review by tigroferoce on Thursday October 03, 2013 @04:29PM (#6491804)
Completely agree. I've been into computer science research for few years and not knowing who accepted the papers and which were the reviews always bugged me.

I've always advocated that the reviews behind an accepted paper SHOULD all be public and publicly distributed along with the paper.

Re:The Arsenic DNA paper exposes flaws in peer review by thaumasiotes on Thursday October 03, 2013 @11:38PM (#6493587)
This is definitely a benefit to the readers, and potentially (though not definitely) to the community as a whole. However, it's much worse for the reviewers.

I hear so much about how researchers provide papers to journals for free, and review papers for journals for free, and then the journals (are presumably evil to) charge high fees to anyone who wants to read the papers. How much can we pile on the reviewers before they stop doing it?

Re:The Arsenic DNA paper exposes flaws in peer review by tigroferoce on Friday October 04, 2013 @05:51AM (#6494531)
Good point. But it is a key point that making reviews is part of your job as researched and you MUST take it very seriously. I received so many reviews poorly written, just few lines saying "OK for me". I really think that this way of reviewing is wrong. Reviewers should take care in their job and they should be ready to face a public discussion on why they decided to accept a paper.

Note that IMHO is OK to be anonymous while refusing a paper because otherwise a reviewer may be put under lot of pressure.

Re:The Arsenic DNA paper exposes flaws in peer review by pizza_boy on Thursday October 03, 2013 @04:25PM (#6491765)
> it would also give some recognition to the work that goes into reviewing a paper

I think you're spot on with this comment. That's exactly what we're building at publons.com. Check it out.

Re:The Arsenic DNA paper exposes flaws in peer review by zokier on Thursday October 03, 2013 @04:29PM (#6491807)
Maybe the general idea should shift towards papers being more of a opening for discussion, where as reviewers would be privileged to get "first posts".
Re:The Arsenic DNA paper exposes flaws in peer review by arjunnarayan on Thursday October 03, 2013 @04:39PM (#6491857)
As any scientist will tell you, this is in fact how we (as academics) do view publication. They are the first step, and definitely not the end goal, of academia.
Re:The Arsenic DNA paper exposes flaws in peer review by jeltz on Thursday October 03, 2013 @05:50PM (#6492376)
There are huge differences between different fields of science. In some publication has become the end goal.
Re:The Arsenic DNA paper exposes flaws in peer review by snowwrestler on Thursday October 03, 2013 @08:40PM (#6493080)
There is a transparent review process for the useful life of a published work. Other scientists can read it, test its conclusions themselves, and publish their own papers supporting or critiquing the original work.

If a paper claims a significant result, it will eventually get tested.

If it claims an insignificant result, it might never get tested. That's ok, little harm done because it is insignificant.

Re:The Arsenic DNA paper exposes flaws in peer review by SoftwareMaven on Friday October 04, 2013 @12:10AM (#6493673)
Contrast that, though, with the extremely sad state of nutrition, where an absolute difference of 0.25% gets trumpeted immediately to the media at "200% more likely", even though the effects are measured through observational studies with no proper controls. A field where who you are matters much more than the data you're presenting (and it seems reasonably accepted to throw out data that disagrees with your pet hypothesis).

I recognize that, in many ways, physics has it easy: there are few sciences that reduce as well as it does and that offer ways to isolate variables like it does (even if it requires multi-billion dollar machines and years to make that happen). But it is still a shining exemplar of gradual evidence building theories to find more evidence, and there are many, many fields that aren't anywhere near that.

Re:The Arsenic DNA paper exposes flaws in peer review by cma on Thursday October 03, 2013 @05:19PM (#6492145)
Physics is an area where almost all papers have pre-prints available freely on arXive.org... sort of going along with the author's point?
Re:The Arsenic DNA paper exposes flaws in peer review by DaveWalk on Friday October 04, 2013 @10:44AM (#6495629)
In the biomedical sciences (where the original author is speaking from), there is a dichotomy: yes, most researchers don't believe that a single publication is the final word on a topic and that in theory anything could be retracted or modified, but in practice that's how the system works. Publications--both how many you have and where they are--are the prime metric for promotions, tenure, and overall success. Furthermore, when applying for funding your publication record is taken into account, as well as your promotions, tenure, etc. It is currency in the life of an investigator.

Not to mention that research is all about who comes in first; the incentive to validate published work is by and large not there. I recall a Nature article on data reproducibility: only 11% of the TOP, PEER-REVIEWED PAPERS were able to be reproduced[1].

[1] http://www.nature.com/nature/journal/v483/n7391/full/483531a...

Re:The Arsenic DNA paper exposes flaws in peer review by utopkara on Thursday October 03, 2013 @11:52PM (#6493620)
So true! The health of scientific literature is not in the articles that claim things, but in the follow-up articles that verify and/or disprove.

Skepticism is at the very core of science.

Partly to blame are the "non-fiction/science&nature" book category, and the journalist/writer cross-breed who chain-write in newspapers, magazines and churn out books from their articles every year.

At first, the really good ones were a welcome addition to Science literature, for bringing the science to our daily lives. But then they became part of the "where it bleeds, it leads" culture, sans proper journalism and sans scientific thought.

Scientists may make fun of how journal articles are covered in newspapers and blogs, but the negative effects of such coverage are quite real.

Re:The Arsenic DNA paper exposes flaws in peer review by narrator on Thursday October 03, 2013 @05:31PM (#6492241)
The converse also holds true. Just because it isn't published doesn't mean it isn't science. Lots of trade secret processes and such fall under this.
Re:The Arsenic DNA paper exposes flaws in peer review by colechristensen on Thursday October 03, 2013 @04:10PM (#6491645)
You're absolutely correct, scientists don't suffer from any misconception that publishing equals truth. The general public, on the other hand, has little understanding of this.
Re:The Arsenic DNA paper exposes flaws in peer review by GrumpySimon on Thursday October 03, 2013 @10:40PM (#6493423)
Absolutely. I see peer review as a way of - hopefully - filtering out the most egregious crap, and fixing the worst errors in papers while rounding them off.

I feel all these debates are kind of pointless, because no-one I know treats a published article as gospel. We make up our own minds about the paper.

Re:The Arsenic DNA paper exposes flaws in peer review by talmand on Thursday October 03, 2013 @05:11PM (#6492079)
Nor do the reporters and/or editorials that cite such peer-reviewed papers that agree with their opinions.

Wait, maybe they do understand but they also understand that the general public doesn't understand.

Re:The Arsenic DNA paper exposes flaws in peer review by GrumpySimon on Thursday October 03, 2013 @10:43PM (#6493433)
I think this is an artefact of how science is taught: In 1953 Watson and Crick published a paper on the structure of DNA. Everyone smiled and nodded.

In my experience it's more like: Most scientists ignored it. Others were interested but had too much teaching to pay attention to it. A few people got pissed off and tried to prove it wrong. A few other people got excited and tried to build on it.

Re:The Arsenic DNA paper exposes flaws in peer review by haproxy on Friday October 04, 2013 @09:13AM (#6495107)