Except no one can see you did it, so it doesn't make you look like a jerk. A random downvote here and there shouldn't bother anyone. It's probably better off if we just realise this and never talk about it.

Depends on context. I'm at +149 right now on the top-level comment, so one upvote that turned into a downvote isn't really a big deal. It could be pretty annoying if you've worked hard on a long comment and then it goes from 0 to -1 instead of +1 because of a fat-fingered downvote.