If users really shouldn't care about their karma, would the site be served better by not making karma visible at all, for users or for comments? You could keep upvotes and downvotes, and even let users know their own karma and how their own comments are rated (so they know when their contributions are being rewarded), but their would be no high-score table of top users, and users would only be able to guess at a post or a comments karma based on where it appears on the page.
This would take away the motivation some people have to contribute high-quality comments - I'm not sure whether the net result would be positive or negative.
Part of this discussion needs to include a serious discussion of what this site's core goals are. If this site were a web startup that needed to get high traffic, than a game-like system that gets posters addicted to writing comments to try and boost their karma makes sense. If the site is more about making sure it has the highest quality content possible for its original niche target audience, than that would make this approach less appropriate.
I have learned to stay away from political discussions. Ouch!
I now appreciate down votes and see it as a way for the community to 'teach me' (not in a negative way par se but as a guiding/helping hand) in the simplest way possible.
:)