I think most posts receive very few votes in any direction. You may be merely looking at a few downvoters, which I feel is a different population than HN at large, since many people can’t even downvote.