I have for many years now a small search engine project in my free-time pipeline, but I'm before crawling even and I intend to sit for searching part after some of that.
Eventually all of these projects will be highlighted on Awesome F/OSS (https://awsmfoss.com), but for now I'm just going to dump my bookmarks here for other people, since I'm leaving awesome projects out:
Search Engines
AWS OpenSearch https://github.com/opensearch-project/OpenSearch
https://github.com/opensearch-project/OpenSearch-Dashboards
https://github.com/opensearch-project/perftop
https://github.com/go-ego/riot
https://groonga.org/ https://github.com/groonga/groonga
https://github.com/meilisearch/MeiliSearch
https://github.com/mosuka/bayard
https://github.com/nezaboodka/nevod
https://github.com/searx/searx
https://github.com/stryku/okon
https://github.com/toshi-search/Toshi
https://github.com/typesense/typesense
https://github.com/valeriansaliou/sonic
Algolia
https://github.com/marconi1992/algolite
https://github.com/quickwit-inc/quickwit
https://github.com/prabhatsharma/zinc
phalanx https://github.com/blugelabs/bluge https://github.com/mosuka/phalanx https://github.com/mosuka/blast
ManticoreSearch
https://github.com/manticoresoftware/manticoresearch
https://github.com/manticoresoftware/docker
https://manticoresearch.com/blog/manticore-alternative-to-el...
https://manual.manticoresearch.com/Introduction
https://forum.manticoresearch.com/t/manticore-search-cheatsh...
https://forum.manticoresearch.com/
Whoosh https://whoosh.readthedocs.io/en/latest/
https://pypi.org/project/Whoosh/
lyra https://github.com/nearform/lyra
https://nearform.github.io/lyra/
https://github.com/LyraSearch/lyra
flexsearch
https://github.com/nextapps-de/flexsearch#performance-benchm...
Lucene
https://github.com/apache/lucene
ZincSearch
Solr
https://solr.apache.org/operator/
https://solr.apache.org/guide/solr/latest/getting-started/so...
https://github.com/apache/solr
https://solr.apache.org/guide/solr/latest/deployment-guide/s...
Konnu https://gitlab.com/shadowislord/konnu
Quickwit QuickWit + Clickhouse
https://clickhouse.com/docs/en/guides/developer/full-text-se...
https://clickhouse.com/docs/en/sql-reference/functions/strin...
There is no way I can get to running all of these (this project was supposed to be quick!!), but I will run the ones I noted earlier, and probably manticore too since it was high on my list since it's quite polished looking.
The list has value on its own, especially if you maintain it.
Are you aware of any that can be used client side like Lyra and supports faceted search?
I've been looking for a solution and cannot find it, even an algorithm and/or a data structure can be helpful. I attempted coming up with a solution myself but ended up with frustration when it came to making the facets dynamic and update as other filters are applied.
I read a couple of papers and one stood out [0], which introduces category theory as a solution to faceted filtering. I understood it in theory and it was still does not seem straight forward to implement but I haven't attempted yet.
0. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5145200/#!po=28...
https://lunrjs.com/docs/index.html
There are some others but I can't find them at this moment -- a bunch of the other projects I find are somewhat abandoned, lunr is actually on my list of things to use (because it makes the most sense to just ship a pre-built index with the first like... 5 letters maybe of typeahead, no matter how fast the backend is)
I don't understand this comment. Why would you search something that *isn't*, in some senses, a repository of information? I would say almost every website needs to have search in some sense, and it's *because* sites function as a repository of information that they need this search. Think about e.g. Stripe's documentation, or Github's repository / code search. HN is also another great example—I search for stories or comments all the time to try and remember something I read about recently or heard about last week, but couldn't quite remember. I'm hard-pressed to think of a web site I use regularly that *shouldn't* have full-text search, if I'm being honest.
The scale of a documentation site is a very different problem -- you can brute force it in ways that you can't at larger scales.
I agree that HN would be a case of the large repository, but even then what most people want out of HN search is pretty simple/basic keyword search. I think a decent non-frustrating HN search feature could be very basic and get by without most of the advanced features/rabbit holes available in search.
Basically I think most apps fall into the lighter search use case -- command palettes, search inside of apps with a small scale of information, etc.
My comment wasn't that apps shouldn't have full text search -- it was that most that have full text search don't need complex full text search with all the bells and whistles that lucene and other serious search engines provide. These up-and-comers might be enough for a bunch of apps for which search is not the main feature.
Just gooling site:foo.com/baz <query> almost always produces better results.
Waiting for meilisearch to ingest documents right now and the Show HN is going up.
https://news.ycombinator.com/item?id=33321268
Maybe I should have used their batch thing instead.
Right now I've got the site going on just Postgres FTS + trigram and it's pretty darn fast, looks like I need to test sonic too.
Going to burn some midnight oil (in my timezone, anyway) and get it out -- though sonic isn't implemented yet!
Anyway to make this comment useful to people, here's my short list of engines that I want to run in parallel:
- MeiliSearch (https://github.com/meilisearch/MeiliSearch)
- TypeSense (https://github.com/typesense/typesense)
- Lyra (https://github.com/LyraSearch/lyra)
- OpenSearch (https://github.com/opensearch-project/OpenSearch)
- ZincSearch (https://github.com/prabhatsharma/zinc)
- Sonic (https://github.com/valeriansaliou/sonic)
There isn't enough out there comparing all these for the simple typical fuzzy search/search box usecase, so I'm adapting a little podcast search site I made to try and use all of these at the same time. So far only Postgres though, will try and add Meilisearch today and post it!
Like other people are pointing out, most of these engines won't have all the features of ES (or more accurately Lucene) but I am pretty convinced that most of the time it doesn't actually matter and if someone is searching on your site excessively maybe there's a problem with your UX (unless you're a search engine or repository of information).
[0]: https://supabase.com/blog/postgres-full-text-search-vs-the-r...