I agree with you. The only times i have wanted to downvote was to mark some comment as spam and/or troll. Given that we have a 'flag' link on comments, i don't see a need for downvote, except to undo a upvote.