If you downvote a comment, the link to undo the vote is actually labeled "undown", not "unvote" (upvoted comments have the "unvote" link).

At the top of the comment, you should see something like:

  username 1 hour ago | undown [-]
                          ^
                          click here

Are you sure about that? I can unflag something I flag but I don't see any place where I can unvote. Unless it's karma gated? Though I would have figured I would have access to all the features by now...