News for hackers. Stuff that matters.
Re:EU Open Web Search project kicked off by arjenpdevries on Tuesday September 20, 2022 @04:31PM (#32917375)
We like Federated search, we like decentralized search, and even P2P search; we are trying to find a good mix, and decided to get started rather than wait! Exciting times.
Re:EU Open Web Search project kicked off by marginalia_nu on Tuesday September 20, 2022 @06:50PM (#32918831)
What are the benefits from this?

I'm not trying to be dismissive, it's just my feeling from working on search.marginalia.nu is that nearly every aspect of search benefits from locality, not only is the full crawl-set instrumental in determining both domain rankings and relevance signals on a term-level such as anchor tag keywords; but the way an inverted index is typically set up is extremely disk cache friendly where the access pattern for checking the first document warms up the cache for the other queries, but that discount obviously only exists when it's the same cache.

Re:EU Open Web Search project kicked off by boyter on Tuesday September 20, 2022 @08:40PM (#32919656)
You could get people creating indexes with love such as your own. marginalia could become the de-facto index for long form content. However you probably arent that interested in running the best pokemon website, so someone else could do that.

Enough people add domain specific search endpoints, with perhaps a taxonomy to say "hey send those sort of queries over here" and you have a compelling engine that self heals should someone stop running things, or starts spamming.

Re:EU Open Web Search project kicked off by arjenpdevries on Wednesday September 21, 2022 @09:46AM (#32925074)
Yes, that is an advantage.

You can also integrate search results for which you cannot have the index, like social media APIs, another reason.

You could also mix and match search results from various topic-oriented indices. That's a research question, whether that is really better than building one unified one. But we think it is the way to bring index fragments to the edge, with the obvious privacy advantages.

Re:EU Open Web Search project kicked off by hkt on Tuesday September 20, 2022 @05:14PM (#32917872)
I would love to be able to run a node that mirrors part or all of an index like this, and to let people query it - a bit like https://torrents-csv.ml/#/

Good luck! I'll be watching your progress and cheering you all on!