I don't know if it's still true, but it used to be the case that if your reply to a comment your downvote to that comment does not count. It was a pretty clever mechanism in that you could be certain any downvotes didn't come from people replying to you.