Kagi (the search engine) has Small Web※, which (I assume) shows only results from the long tail and excludes popular sites (they use TinyGem search index for this).

※https://blog.kagi.com/small-web

Oh this looks really nice. Thanks for sharing!

I mean put adblocks into crazy mode, blacklist anything related to Google and Meta-crap by default, turn off JS and probably CSS too and you are almost there, maybe apart from blinking text and weirdly stretched bitmap backgrounds

Wasn't there a site that showed search results but skipped the top N results, so you ended up seeing a lot more interesting smaller sites? Can't remember details and a quick search (hah) didn't find it. Pretty sure it was linked on HN a year or two ago...

Edit - I think this was it, but looks like it needs an account: https://millionshort.com/

> so you ended up seeing a lot more interesting smaller sites?

A lot of smaller sites just aren't indexed any more, it's a common complaint you read here...

"Smaller site, we don't use adwords, we're getting crawled 500x/day which is 40% of our bandwidth, still not indexed after 6 months"

If a search engine were to penalize ads, I bet a lot of those old sites would surface. The old web was an "amateur" web, people created sites to share content they were excited about, not to monetize it (or to allow a platform to monetize it). If search engines also make most of their money from advertising, they have a big conflict of interest. We need an open source search engine owned by a non-profit organization following the model of Signal.

> If a search engine were to penalize ads, I bet a lot of those old sites would surface.

Maybe using an ad blocker to build a list of "websites that serve ads" (in addition to domains those ads are served from), and using that list to filter out results from an existing search engine API (like Duck Duck Go) would get you part of the way. But you'd probably need to search really far before you start finding sites that don't serve ads.

Kagi partially does that. I lt shows how many trackers a result has on its results page and lets you boost or downgrade domains in your search results.

I would use a search engine that only indexed sites that have zero ads, tracking or referral links.

Not sure how useful it'd be, but I would use it!

It would probably be like Google circa 1999-2000

I would use that too, for sure! I kind of feel like experimenting with this now. I have no idea how doable it is to just "start scraping the web" though. uBlock Origin is open source, so I guess that could be used to help filter out sites that serve ads.

I find this CSE quite usefull for finding old websites https://www.oldestsearch.com/

All you need is a reliable way to detect cookie popups.