there are tons of ways to prevent that by weighting votes and only taking a random sample. SEO is basically a war where SEOs find a way to exploit Google's algorithm and Google tries to respond.
Backlinks were Google's original solution to keyword spam that made other search engines suck, they didn't expect people would create entire sites or pay other websites to send links to them so they could rank higher. Backlinks were themselves considered to be "votes" for quality similar to citations in academia. The problem is the barrier to entry to make a site and link to it is high, so SEOs themselves are often the only ones to do it. Adding some sort of voting mechanism would allow actual readers to give an opinion on quality, although you'd probably see SEOs pay people to vote on their link via a VPN switching IPs, so only people with established Google accounts could be allowed to vote. Obviously its a huge challenge, which is why Google is worth hundreds of billions for even partially solving it
Helps for both search users and the platform. Like a free moderation service. Why have an employee too check if these pages should be removed if a crowd of tracked users can do it.
I wonder where is my view wrong because I wonder why isn't there such system yet...
In order to detect malicious activity you'd have to use a bunch of other signals, and also to use the downvotes with some suspicion, so you're back at looking content quality, reputation, page speed, and many other signals. Which sounds very similar to status quo.
Personally I'd like to see some sort of explicit input. If not used collectively, at least to customize my own results, like it happens on YT/Netflix/Instagram/etc - e.g. "never show results again from this site", or "I dislike this type of content".
You forgot to add "Because clever posters use bots to give an early upvote boost to their posts, and downvote competing posts" And the more eyeballs on the site, the more things will be manipulated this way by bad actors.
For all its flaws maybe becoming like reddit it what search needs. AI generated spam will likely always be better at gaming a search algorithm than a human. Human moderation has its flaws but at least we're very familiar with them - they might be worth dealing with if it keeps the internet vaguely usable.
Beyond that a once popular website or producer says or does something that offends the sensibilities of the internet crowd and then everyone goes and down votes their results because it violates the current ziegiest