Link aggregators are also very much part of the problem. Besides the astroturfing, they tend to favor the lowest common denominator in content. Popular vote just doesn't seem to be a good way of surfacing quality content, especially not with the social incentive of winning internet acclaim for submitting a good result.
HN does surface the odd gem, but why can't I pick one of those and say "Hey, this topic interests me, show me more like this"? I'm currently working on build something to that effect, and so far it seems at least reasonably possible.
RSS was good in one way, but it also sort of turns everything it comes into contact with into a blog.
Why not start it? One of the beauties of reddit subgroups is that not only do you find random very specific ones, but if you don't, you can start your own.
Oh, wait
I use search for information, but I often have to dig quite deep to find something worth clicking on or reading. And that's true for Google and DDG (which I mostly use).
When it's raw, basic knowledge, then I'm using !wiki or !mdn etc. in DDG to search those sites directly. Note: Wikipedia and Mozilla Developer Network are two of my favorite sites.
But when it's something more subtle and specialized that I'm not very familiar with then often my only hope of finding something useful is to search via aggregators and forums like HN, Reddit etc. because then I can gauge the quality of the links via the discussion around it and the people who post them. Or in some cases I know about a specialist or journalist and can use their name to filter out spam.
Discovering these kind of things via search is really hard, especially if their authors are not known. It's the spammers and low effort creators that get rewarded the most, partly because of search engines, partly because of visitors who are content with that kind of quality.
I wish I knew some decent curators or magazines, but most of the stuff I come across is just too far from what I‘m looking for. Search engines don’t help much here either.
Some of these raw looking indie games have turned out to be gems. Thinking of Faster than Light, Loop Hero etc. It's hard to gauge the game-play from a quick look. Not even "similar to" searches help too much. It's not really about the money spent, but rather about the time both finding and then playing. I wish I had a list of curators that I really dig. So far, hit and miss.
Has the web ever been much beyond that? Have people ever really been just hitting "give me a random web page about X"?
Of course search engines are very useful, just that these conversations about content and "the web", I feel like there's not much focus on how so much traffic is guided through link aggregation.