I think of downvotes as opposite of upvotes. If you agree with the gist of a comment and find it informative or useful, you upvote it. If you really disagree, you downvote it (and if possible, drop a line to say why you disagree). If I'm wrong and that's not the way things are being done here, I apologize from OP.

I think another way could be to decrease a users karma by 1 for every 5 downvotes he makes... and also limit the number of downvotes a user can make in a day.