I believe there is a 'downvote' button, but it's only available once you've scored above a certain karma threshold.

Also, while you don't know how often something has been upvoted, a comment with a higher score will be at the top of the page, and you'll see it first and be more likely to give it an additional upvote.