I would love to have some kind of stack overflow type system for commenting on papers. If you find a bug in some paper chances are someone else will benefit from the knowledge. Oh, the authors didn't report some key step in the algorithm and you figure it out? Pay it forward and leave a comment! Crowd sourced errata, if you will.

Hell, integrate with GitHub so you can link to implementations, etc.

The fact that reader comments are controversial (about as many people are opposed to this as in favor, see graph in article) suggests that it's a great idea, and that the design of the reputation system is critical.

Perhaps they can semi-hide the comments during an introductory phase where they develop the reputation system. A well-crafted reputation system could eventually take on a role resembling peer review, but should be approached very carefully.

I'm not an academic, but I think you'll find many academics would be against this. Things would get more complicated because there would obviously need to be a vetting process on who could comment. Things shouldn't be a popularity contest, so I think a reputation system would be counterproductive.

>Things shouldn't be a popularity contest

You clearly are not an academic. Publishing papers is a popularity contest.

But it's a different type of popularity contest. Commenting on publications is virtually non-existent. It's not that there is no venue of commenting, we do have plenty of academic conferences; but the amount of comments received in the conferences is virtually none. People do ask questions, but seldom give constructive (be it positive or negative) comments.

I don't think this is true in general (though it might be so in the field you are working in). I've received many constructive comments from colleagues, and this has often lead to subsequent collaboration.

Pubpeer?

Are you an academic? Because peer review is almost always (at least in CS) blind, with authors anonymous to the reviewers.

Well, I find CS venues (conferences and journals) to be about 50/50 in terms of blind or not. Also, while it will certainly vary subfield to subfield, once you are familiar enough with an area you can pretty confidently guess what group (if not what exact authors) are behind any given paper. I bet it would be pretty easy to cook up a automated paper un-anomymizer based on the body of published work out there already.

In my field of laser physics, there are some really, well known lasers, so if your paper talks about this or that facility and cites this "monumental work by X group" you can easily gather who is the author. Not to mention (yes heaarsay) of at least one person in my group who refereed a paper, harshly criticizing it paper only to have the editor come down and say "accept this paper anyway." Academia has some good parts but it's as political as anything, if not more so.

Reputation systems are nice for providing a carnal motivation, but moderation is likely going to be necessary to maintain a high quality density. Crowd-sourced moderation a la slashdot is a reasonable place to start. I think there's a lot of room for improvement in this space, though. An academic publishing

I hope that in the future, something like ArXiv with a discussion system will be built on a fully distributed network a la IPFS. That's been one of my most ambitious dream projects since my third year of grad school. I haven't made any progress towards it. I think such a system could stave off some of the decline in academia. I'm glad that ArXiv is headed in that direction. They're the experts.

I don't think this is a useful thing to have. There is no clear need for it, and it would likely reinforce group think a la HN or reddit.