First, the price. The last paper I reviewed took me about 10 hours. I charge $200/hour. Should I charge $2000 for my time? If I'm paid by the hour, who tells me when to stop? If I'm paid a flat fee, how much is that fee?
I have seen a reviewer write something like "I read the paper twice. It seems good. Publish it." Does that get the same as someone who spends 10 hours for a detailed critique?
Second, does payment constitute work-for-hire? Does the publisher and/or reviewer need to pay taxes? Might some reviewers have a contract which prevents them from taking on part-time work? What if the reviewer is overseas? Do tax agencies from two different countries need to be involved?
Third, most people expect to publish, and get their papers reviewed. If a single author has three reviewers, then that author should expect to do about three reviews. This is an imperfect system which rewards publishing rather than reviewers, but there are also some compensations to the reviewer.
Moreover, I recall one story - I don't know the validity of it, but it has a ring of truth - where a daycare got annoyed with parents who picked up the children late. They decided to fine parents should that happen. As a result, children were picked up late more often because the fine was interpreted as a fee for extra daycare, and no longer social chastisement.
I suspect that reviewers are in a similar boat. I'll feel different about it if I get paid a pittance than if I were to do it for free, so long as others do the same for me in the future.
Of the people I meet at a conference, how do I figure out who has been a reviewer for Nature, and how do I verify that claim?
> What's wrong with giving other researchers the ability to rate and comment on a paper.
I do not see nothing inherently bad with this (provided very good moderation tools are in place). However...
* > I would argue it works just as well a this journal reputation system *
As a journal reputation system, possibly; although I find that experts have often very different taste compared to say general Hacker News public -- for instance, machine learning has a very high "coolness factor" these days compared to other areas of computer science, but it would be unwise to make a system where a weaker paper on machine learning gets vastly more credit than a better paper on say approximation algorithms.
However, an important part of the journal submission process is the fact that the editors ask the members of the academic community for spending several days on the paper, noting major and minor details and checking the validity of the proof.
Since the reviewers do their work for free, there is no reason why there cannot be an open system that replicates the same process; still, just "adding a comment system to arXiv" will probably not be enough, as people spend rarely that much time on comments (15 minutes tops, not entire days).