If someone downvotes me, I would indeed like to know why.

If someone downvotes me because they disagree with me, I would prefer they state their disagreement rather than downvote.

If someone downvotes me because they think I'm being rude, I would like to know what it is about my comment that they consider rude.

If someone downvotes me for being an Apple/Google/Microsoft fanboy, I would like the opportunity to disabuse them of that notion.

If someone downvotes me because my comment is incorrect, I would like to know what the truth is.

Downvoting is sometimes used as a lazy way of saying "This is a poor comment" (I know, I do it myself sometimes when I am tired). But without saying why a comment is poor, it is itself a poor form of feedback.

I might come back to a comment I wrote the day before, and in a different frame of mind be able to see why it has been downvoted. But when I can't see why, it is frustrating.

I don't really care about my HN karma. It's just a number that tends to go up sporadically.