I understand that often often times people will use downvotes as a substitute for disagreeing, despite that being explicitly against its true function. Ultimately, you're complaining about fictitious karma points.
If you are a regular contributor who follows the rules, over time your karma will accrue and this can be a fuzzy way to weigh your importance to the community. The danger of valuing karma too much will cause the system to be gamed by marketers and astroturfers (see Reddit for Example A)
Early on with HN pg stated that downvoting for disagreement was proper behaviour on HN. My opinion is that's wrong but, in contrast to other fora, downvoting to disagree is thus explicitly a part of proper behaviour here.
That said, I don't care about the points, it's a matter of social value - if you don't know why people are disagreeing then you can't address that concern or reassess your own position, the downvoted adds no value whilst a comment may.
It might not help, of course.
Moreover, without vote visibility you can't tell if the reason given by one person is supported by others.