I’ve never used it, but the way others do, here and on Reddit, I think of it as the “disagree” button, rather than “downvote”.

It can be frustrating when people use it in lieu of a comment, though. It means that, of all the arguments at their disposal, the person decided to expend only the minimal effort required to produce something that still qualifies as a reply. The communication via binary responses is a great illustration of the decay of modern discourse, if you believe in that sort of thing.