Wouldn't that present the same issues that removing the "dislike" counter on YouTube videos did?

For example, that video by The Verge where that guy builds a PC in the stupidest way - it's got thousands of likes for some reason but 10s of thousands of dislikes because it's not the way you should build a PC

Now the dislike counter is gone, those who see the video think "oh this has thousands of likes. It must be good".

Downvoting does attract idiots and pile-ons, but it does serve a purpose as a a lose indicator of content/comment quality