Completely agree. The problem is that the downvote means too many things at the same time (Spam AND Disagree AND Not-adding-to-the-convo)

HN rules suggest that one shouldn't downvote to disagree, but it's not what's happening.

One solution is to keep the upvote button and replace the downvote one with a link, e.g. "Report" where you'd pick the reason.