I remember, in the 90's, I typed "hexdit" in altavista and spent the time visiting exactly 100 websites to find exactly the hex editor I was searching for. I remember the number exactly because it was in the last position (10) of the last page and there were 10 pages.

Ugh. I distinctly remember a dinner party in 1993-94 where we talked about how amazing the web was and how impossible it was to find anything. We discussed registered keywords a la AOL, and I’m sure someone suggested search engines would eventually solve it, but I certainly wouldn’t have wagered significant money on that based on Altavista.

When I discovered Google (I think it was running at beta.stanford.edu at the time) it was just short of a miracle.

This is a big part of the problem. The incentives.

There is no blank space to fill with something useful.

Instead everything is filled with SEO driven, ad riddled, bullshit.

And creating something useful is more cost and effort than creating more SEO driven crap.

And even if you do create something useful, you’d then need to focus on SEO in order to compete.

There’s no way this problem isn’t solvable. But google seems to lack any motivation to solve it.

Around 20 years ago I was trying to intersect data sets on two machines without transferring the entire set across the wire. I was slowly, painfully trying to invent Bloom Filters, because none of my searches turned up Bloom Filters. The company folded before I could well and truly embarrass myself.