News for hackers. Stuff that matters.
Re:A look at search engines with their own indexes (2021) by jfoster on Tuesday June 21, 2022 @04:00AM (#31821451)
For most software, there is an open source attempt aimed at producing something more or less equivalent. Is there any such thing trying to attempt the same level of scalability as Google or Bing?

I'm aware of Apache Lucene, Elasticsearch, etc. The thing is, those all seem aimed toward indexing a lot fewer orders of magnitude less stuff than Google does.

I'm guessing that perhaps anyone capable & attempting to implement such a thing just automatically gets fast-tracked into a job at Google (or similar company). Is that why it doesn't exist?

Re:A look at search engines with their own indexes (2021) by native_samples on Tuesday June 21, 2022 @04:49AM (#31821778)
How would you even test that? Open source projects typically have a budget of near zero. There are lots of things that the open source approach can't handle but commercial development can, large scale search engines are one.
Re:A look at search engines with their own indexes (2021) by marginalia_nu on Tuesday June 21, 2022 @05:00AM (#31821844)
I guess hypothetically you could go for a wikimedia-style model and set up a foundation that operates the thing, but that would require a lot of funding, which would require decent utility up front. Vicious circle there.

I'm struggling with how to run even a small scale search engine as an open source project. I haven't really found any good projects to model for how to go about an open development process for a system that actually requires decent hardware to run, so right now it's just me developing in public, which is fine I guess, but not much different from how it was before I open sourced it.

Re:A look at search engines with their own indexes (2021) by psychoslave on Tuesday June 21, 2022 @08:53AM (#31823292)
Re:A look at search engines with their own indexes (2021) by marginalia_nu on Tuesday June 21, 2022 @09:43AM (#31823857)
From what I understand, they basically built a search function for wikipedia (and related projects). A noble goal since their own search function is kinda shit, but it was never really an internet search engine.
Re:A look at search engines with their own indexes (2021) by native_samples on Tuesday June 21, 2022 @02:32PM (#31827370)
Well, what you could try is an engine that's designed to crawl specific niches and then allowing people to run their own instances. So you don't run any hardware yourself. You just let other people provide it - they might find a way to make money from it, or their community might find ways to pay for it anyway. It's not necessarily true that every search engine has to search the entire web.
Re:A look at search engines with their own indexes (2021) by marginalia_nu on Tuesday June 21, 2022 @03:51PM (#31828335)
The problem with this is that you get significant benefits from a larger crawling corpus, even if you index just a small portion of it, the rest will inform that portion.

That's a real problem that is ultimately hard to get around. Like my index is fairly small as it is, but even so it requires more hardware than say a student could afford. Like it's not enough that I'm dependent on external funding, but it's still not something you slap together for fun and then get bored of.

Re:A look at search engines with their own indexes (2021) by jfoster on Tuesday June 21, 2022 @05:11AM (#31821900)
If you're going that far, I guess you might as well set up a commercial search engine or SEO tools business off the back of it. At which point, the thinking may shift away from open source toward something proprietary, unless there were some other substantial differentiators.
Re:A look at search engines with their own indexes (2021) by oever on Tuesday June 21, 2022 @11:03AM (#31824740)
> Let’s create a better way to search the internet

> We want to make searching the internet open and transparent. Our goal is to provide a wide range of independent and free options for navigating the web. This is relevant for all organizations and people who want to boost Europe’s digital sovereignty, a greater variety of search engines, and independent search results. We can make this possible by cooperating with existing data centres and by using open source principles and public moderation. We may not be able to do it as individuals, but we can do it together.

https://opensearchfoundation.org/

Re:A look at search engines with their own indexes (2021) by pyronik19 on Tuesday June 21, 2022 @05:48PM (#31829479)
Will it censor "vaccine misinformation", "election misinformation", "russian disinformation" and all the other mainstream topics that google actively suppresses in favor of the narrative or is your moniker of "open" just mean "open to the ideas we support".