Depends on context. I'm at +149 right now on the top-level comment, so one upvote that turned into a downvote isn't really a big deal. It could be pretty annoying if you've worked hard on a long comment and then it goes from 0 to -1 instead of +1 because of a fat-fingered downvote.