Exactly. One possible fix would be that you don't actually get to see a downvote arrow on anything unless you reply to it first.

Separately, I've had thoughtfully written responses flagged for what is presumably disagreement. So may be the "flag" button needs a dropdown and the downvote button not showing up until you reply would be a good combination solution.