A rating system would generally become an indicator of other academics' level of agreement with a paper, rather than an indication that it passed a quality and significance threshold. Citation ranking schemes already exist, but the citation counts are heavily influenced by academics being unable to NOT mention a dissenting view because it was recently published in a prominent journal. And I'm not sure examples like Amazon book ratings, HN karma or professionally-optimized PageRank can really be claimed with a straight face to be superior for curating quality content than good old fashioned peer review.

I do think open commenting systems would be a boon to academic discourse though.