The content on the sites I visit is created by humans. Until automation genuinely overtakes us, I'm not ready to accept at face value the scale of the internet has grown so large that humans couldn't tackle the problem.
All I can say is, good luck with your human curation startup.
If there were orders magnitude more pages than humans, I'd agree. But I'd also ask: Who created them all?
It's not easy to quantify the amount of useful content on the internet. The 2bn figure above seems to stem from registered domains, and depending who you ask [1][2] around three quarters of them are "inactive" (e.g. landing page for a parked domain).
At the other end of the spectrum, Google's index surpassed 130 trillion pages four years ago [3]. Point in favour of my opponent!
If everyone connected to the internet indexed one page a day over the course of their lifetime we might just about do it. And anyone creating a new page would need to [arrange to] index it themselves.
[1] https://www.internetlivestats.com/total-number-of-websites/
[2] https://hostingtribunal.com/blog/how-many-websites/#:~:text=....
[3] https://searchengineland.com/googles-search-indexes-hits-130...
Also, any setup that allows everybody to be a moderator will be promptly gamed.
What you want is what yahoo used to do - a hand-curated search engine. It worked when the internet was small, but got buried under the eventual avalanche of web sites.
- Using DNS "zone files", the DNS database for TLDs (which are not available for all, but most) show there's circa 200 million domains registered at any moment
- A large percentage of these are parked, i.e. no unique content.
- Many domains are "tasted", i.e. bought, are alive for a few days then disappear, so potentially you waste time crawling them
- Lots of sites are database driven and can result in millions of pages that can be created in a day
- URL rewriting means you can have an almost infinite number of pages on any one site
- Soft 404s and duplicate content can be hard to spot and can waste resources in gathering/removing them
There's paid for resources like Majestic/Ahrefs/Moz that crawl the web to see who's linking to who and they all contain trillions of URLs.
I think the most detrimental fact is that pages often disappear or change, I don't have a recent number but I'm fairly certain there's a 10-15% chance that any link you see this year, will be gone next year. "Link rot". Hard to build a DMOZ style directory on that scale with that problem.
I don't think it is unmanageable, it just needs to be seen from different perspectives and managed by different groups of people.
You only need to choose an algorithm so that the amount of trash is less than you can handle.
2. An algorithm can always be gamed.
You're stuck.