This is the problem. The upvoting-downvoting is a good idea in principle but it does not (and cannot) take human subjectivity into account, which is what makes it fundamentally flawed. By the way this is an extension of the same problem as with silencing opinions or comments online or in real life.