I think it would be neat if downvotes took a point of karma away from the downvoter as well.

That would provide incentive for the downvoting user to comment so that people would have a chance to mod him up to recover it.

Yes, yes, I know "don't pay attention to your karma", but it's one of the primary pieces of information about our posts/comments that we're given by reddit/this system.

The site would function identically if article, comment, and user karma values were all hidden. I wonder what the impact would be on the dialog.