If that was the intended design, then perhaps votes should be hidden from users as well, so that we'd never know if we were voted up or down; but, I think the design is intended to somewhat subtly clue us in about whether our opinions and statements are popular, controversial, or whatever. It's a feedback loop, and most forms of conversation involve such indirect feedback loops in the form of non-verbal communication -- though in real-life, it's heavily augmented by body language, facial expressions, voice tone and inflection, etc. We don't have those luxuries here, and so the only non-verbals we're left with are the up-arrows.
It's one thing to receive a single down-vote, hell you can't please everyone, right? However, when you try in earnest to participate in a conversation, and you receive the equivalent of a "shut up" from the crowd, then you're left wondering: did I miss the entire point? Did I accidentally offend someone? Was I factually incorrect?
For a healthy conversational atmosphere, it helps to resolve these issues, so that we can improve our communication -- and hopefully avoid the need for any downvotes in the future.
Some days, I wish I could down vote down votes.