If you downvote a comment, the link to undo the vote is actually labeled "undown", not "unvote" (upvoted comments have the "unvote" link).

At the top of the comment, you should see something like:

  username 1 hour ago | undown [-]
                          ^
                          click here