I define "karma" as a means of valuing your contributions to the community, and it seems that this ought not be dominated simply by what you contribute in a literal sense, but also in what you contribute by encouraging others by voting up their valuable contributions. If our definitions differ, I can see that being the cause of the disagreement.
For an exploration of similar ideas, take a clicky: http://www.nplusonemag.com/?q=node/473 (I actually think I might have seen here first, but forgotten to save it...)
In other words, if you want to look at this in absolutes and think that when I say that I think it's a good idea to take a given variable into account that I am somehow implying that the variable is free of all bias and in every other way perfect, you are making a mistake. You will never have high confidence for all of your variables when they describe human interaction, but that low confidence doesn't speak to the utility of the variable, it speaks to its importance in the overall calculation.
If you boil it down, my point is that while "do people like what I say?" is a suboptimal scoring criterion, "do most other people like the same things I say I like?" is likely to be significantly worse.