I don't really know how you migrated to the subject of downvoting, but I do not disagree. Never found the idea of "karma" on a forum to be useful for anything other that brigading and collective censorship. I use Harmonic to view HN, which has options yo hide scores and all that, do I don't even see the downvotes, tbh. Safe to assume they are there though, especially when you say things that do not jive with the popular narrative.
Anyway, I have unpopular, but sincerely held opinions - anyone is free to disagree, of course. The point I'm trying to get to is that while I hardly ever downvote, if I did, I would hope that I would also comment on what the issue was.
As a courtesy, it should be downvote + comment, if poss.