We can do better than DuckDuckGo

Re:We can do better than DuckDuckGo by oever on Tuesday November 17, 2020 @06:42PM (#25131397)

Instead of having many bots do inefficient crawling, web sites should publish their own index. Intermediate parties can combine indexes of the sites. Sites that do not provide indexes get less visitors.

[ Parent | on HN ]

Re:We can do better than DuckDuckGo by rakoo on Wednesday November 18, 2020 @04:03AM (#25134610)

That's the best way to fill your search engine with spam. There needs to be a third-party that verifies that the site-provided index is inline with the actual content. At which point said third-party can be a crawler.

[ Parent | on HN ]

Re:We can do better than DuckDuckGo by adrianstoll on Tuesday November 17, 2020 @07:04PM (#25131565)

This is the idea behind sitemaps which have existed forever.

[ Parent | on HN ]

Re:We can do better than DuckDuckGo by oever on Wednesday November 18, 2020 @03:13AM (#25134281)

Sitemaps are lists of urls on a site. They are not a text index.

[ Parent | on HN ]