I upvote replies which are high quality or with whom I'd like to have a conversation. But there are lots of people who I wouldn't want to talk to here: lots of well ackchuallys, poor Bayesian reasoners, etc.

Quite simply, if I were replying in a moron fashion, I'd prefer you left me without the upvote as well, so I'll do the same to you.