Just like normal karma, the only person you are competing with is yourself. I enjoy it when I write a comment that others either appreciate or agree with and think is the "correct" sentiment, and find that downvotes are sometimes interesting as well as I find that I may be in the minority on a viewpoint.

The only thing I find sometimes irritating is that sometimes when I am flip I get more upvotes than when I am being serious, but maybe it's just HN telling me to speak my mind more.