I've often prefaced such replies with "Because sometimes a upvote isn't enough...", and those get downvoted about 40% of the time. Not that I care, I bathe daily in karma points, it's just a tiny bit disheartening.

I would also argue that an upvote is not "I agree with you", but "this contributes to the conversation in a positive manner". I frequently upvote posts with which I disagree. Conversely, a downvote in my book says, "hey, you're kind of being a dick and dragging the conversation down" rather than "I disagree". I mean, I'll downvote something that is just demonstrably wrong, but more often than not it's "quit being a dick".