Personally I think you're taking things too seriously, even though you're right. I enjoyed "The Great Online Game"[1] recently (1 comment), which stages this engagement as play. Although I would like to see my karma go up faster, that's not the point of the play for me. Unfortunately a lot of ludicrous, non-contributing, downvoting schmoes are part of the game. It's ok, it's still- for me- for fun, and I don't feel the need to adjust myself or my strategy of play because of the directed, biased negativity about.

To me, the missing part of the equation is that down-voters don't have any stake in their downvotes (in many systems: Reddit, HN). De-anonymizing these behaviors, is, to me, how we begin to form an awareness of each other, is how we ought to stake ourselves in when we go about saying something is not valueable. We should be able to orient ourselves better online, understand each other, and build our own defensive & observatory systems. The web should be rich in information, & growingly interlinked, and from this we ought to be able to emerge, slowly, over decades, interesting webs of trust, and identifying the negative forces, the downvoters, is a key piece of understanding this web ought be able to encompass.

[1] https://news.ycombinator.com/item?id=27955488