This invites a kind of trolling. Imagine a bunch of people down-vote a comment quickly (justified or not), making it hard to read. Later readers who may have different opinions are discouraged from expressing them (up-voting) because the comment has become so hard to read it may not get read.