In ideal case, yes, but there's nothing to prevent from just downvoting things that are against your personal opinion. Therefore, it would be abused/ignored.

It's not that most of the people are rational and self-critical - many people think that their personal opinions are right and others are nonsense :)

You can't design system on the assumption that people are rational.