I'm struggling with how to run even a small scale search engine as an open source project. I haven't really found any good projects to model for how to go about an open development process for a system that actually requires decent hardware to run, so right now it's just me developing in public, which is fine I guess, but not much different from how it was before I open sourced it.
https://en.wikipedia.org/wiki/Knowledge_Engine_%28Wikimedia_...
https://www.newsweek.com/wikipedia-takes-google-new-transpar...
Spoiler: it failed.
That's a real problem that is ultimately hard to get around. Like my index is fairly small as it is, but even so it requires more hardware than say a student could afford. Like it's not enough that I'm dependent on external funding, but it's still not something you slap together for fun and then get bored of.
> We want to make searching the internet open and transparent. Our goal is to provide a wide range of independent and free options for navigating the web. This is relevant for all organizations and people who want to boost Europe’s digital sovereignty, a greater variety of search engines, and independent search results. We can make this possible by cooperating with existing data centres and by using open source principles and public moderation. We may not be able to do it as individuals, but we can do it together.
I'm aware of Apache Lucene, Elasticsearch, etc. The thing is, those all seem aimed toward indexing a lot fewer orders of magnitude less stuff than Google does.
I'm guessing that perhaps anyone capable & attempting to implement such a thing just automatically gets fast-tracked into a job at Google (or similar company). Is that why it doesn't exist?