If we're going to have any system of upvoting and community-flagging, then that's the outcome. It's like when people point to upvotes and downvotes on Reddit as examples of some sort of pernicious media bias; no, it's an online community where upvotes and downvotes are supposed to serve some sort of valid ranking, and everyone thinks the poster was wrong, or an asshole, or whatever. To complain about the effects of the downvotes is to basically say the downvoters shouldn't be allowed to express their opinion via that available mechanism.