I'd like to see downvotes only used when a comment doesn't add to the conversation, not when the comment gives a wrong answer. Some comments highlight a popular misconception which is useful to the conversation.

I think this is more a UX problem, where sites like Stack Overflow use downvotes to indicate that they disagree with the answer.

Maybe instead of a downvote button it should be a flag or -_- emoji. (edit: apparently HN removes emojis)