If someone makes a quality comment, but it happens to be an unpopular view, and they get voted negative, they are less likely to express their view in the future... and we are the poorer for it. No one wants a monoculture.
If that same person makes the same comment, and people merely upvote the opposing viewpoint, then neither person is being 'discouraged' from contributing. One is just being encouraged more than the other.
I realize it's all just a number and shouldn't matter, but we are human beings and it does... seeing someone else get told "good job" isn't near as bad as someone telling you "bad job"... especially when you thought that you had done something good by contributing a useful viewpoint.
(Obviously none of this applies to trolls/spam and things that are outright false... people shouldn't say things they can't provide evidence for.)
To make a rather tenuous analogy: If you don't like a product, you don't get to take money away from the company that makes that product. The best you can do is give money to their competitor. And to extend that, you can only take money (fines) from some company if they do something illegal. Here doing something 'illegal' is trolling/spamming.
I see a lot of people saying this, but judging by peoples' actions I'm not sure it's true. If I had to guess I'd say that maybe 1/4 (of hackers, a far smaller slice of the general public) of folks really do value a truly vigorous open exchange of ideas and welcome reasoned and thoughtful criticism, with the remaining 3/4 of folks implicitly willing to live within the cozy confines of an ideological fishbowl.
So ideally a lot haters would vote down, and a lot of agree-ers would vote up. And it would balance it out. But this only works if we assume the right amount of people are backing a good idea.
What happens if it's an unpopular but good comment? Then allowing people to downvote would quickly suppress it without enough counteracting "I Agree" upvotes.
So upvotes can be used for more than quality indicator, but a downvote simply makes the system more volatile. If there were just an upvote and no downvote, then the novote will simply be the new downvote. Except the novote will be less volatile.