News for hackers. Stuff that matters.
Re:An alternative to Elasticsearch that runs on a few MBs of... by hardwaresofton on Monday October 24, 2022 @08:57AM (#33316029)
Wow it's weird that this comes up, I'm actually running a site I am going to repost to HN today that I want to use as a testbed for search engines (kind of like an extension to my recent collaboration with supabase[0]).

Right now I've got the site going on just Postgres FTS + trigram and it's pretty darn fast, looks like I need to test sonic too.

Going to burn some midnight oil (in my timezone, anyway) and get it out -- though sonic isn't implemented yet!

Anyway to make this comment useful to people, here's my short list of engines that I want to run in parallel:

- MeiliSearch (https://github.com/meilisearch/MeiliSearch)

- TypeSense (https://github.com/typesense/typesense)

- Lyra (https://github.com/LyraSearch/lyra)

- OpenSearch (https://github.com/opensearch-project/OpenSearch)

- ZincSearch (https://github.com/prabhatsharma/zinc)

- Sonic (https://github.com/valeriansaliou/sonic)

There isn't enough out there comparing all these for the simple typical fuzzy search/search box usecase, so I'm adapting a little podcast search site I made to try and use all of these at the same time. So far only Postgres though, will try and add Meilisearch today and post it!

Like other people are pointing out, most of these engines won't have all the features of ES (or more accurately Lucene) but I am pretty convinced that most of the time it doesn't actually matter and if someone is searching on your site excessively maybe there's a problem with your UX (unless you're a search engine or repository of information).

[0]: https://supabase.com/blog/postgres-full-text-search-vs-the-r...

Re:An alternative to Elasticsearch that runs on a few MBs of... by hawski on Monday October 24, 2022 @10:21AM (#33316977)
Thank you for this comparison. I would also like to know how Bleve Search (https://github.com/blevesearch/bleve) turns out.

I have for many years now a small search engine project in my free-time pipeline, but I'm before crawling even and I intend to sit for searching part after some of that.

Re:An alternative to Elasticsearch that runs on a few MBs of... by hardwaresofton on Monday October 24, 2022 @10:26AM (#33317071)
You're right I should put bleve on there as well. This isn't even the whole list. Toshi (https://github.com/toshi-search/Toshi) is also out there...
Re:An alternative to Elasticsearch that runs on a few MBs of... by snikolaev on Monday October 24, 2022 @10:31AM (#33317148)
If you decide to add Manticore Search to the list feel free to ping me at [email protected] if you need help with preparing the ingestion scripts etc.
Re:An alternative to Elasticsearch that runs on a few MBs of... by hardwaresofton on Monday October 24, 2022 @10:36AM (#33317232)
Oh! Damn it I forgot about manticore -- I had seen it before but forgot to include it.

Eventually all of these projects will be highlighted on Awesome F/OSS (https://awsmfoss.com), but for now I'm just going to dump my bookmarks here for other people, since I'm leaving awesome projects out:

Search Engines

AWS OpenSearch https://github.com/opensearch-project/OpenSearch

https://github.com/opensearch-project/OpenSearch-Dashboards

https://github.com/opensearch-project/perftop

https://github.com/go-ego/riot

https://groonga.org/ https://github.com/groonga/groonga

https://github.com/meilisearch/MeiliSearch

https://github.com/mosuka/bayard

https://github.com/nezaboodka/nevod

https://github.com/searx/searx

https://github.com/stryku/okon

https://github.com/toshi-search/Toshi

https://github.com/typesense/typesense

https://github.com/valeriansaliou/sonic

Algolia

https://github.com/marconi1992/algolite

https://quickwit.io/

https://github.com/quickwit-inc/quickwit

https://docs.meilisearch.com/

https://github.com/prabhatsharma/zinc

phalanx https://github.com/blugelabs/bluge https://github.com/mosuka/phalanx https://github.com/mosuka/blast

ManticoreSearch

https://github.com/manticoresoftware/manticoresearch

https://github.com/manticoresoftware/docker

https://manticoresearch.com/blog/manticore-alternative-to-el...

https://manticoresearch.com/

https://manual.manticoresearch.com/Introduction

https://forum.manticoresearch.com/t/manticore-search-cheatsh...

https://forum.manticoresearch.com/

Whoosh https://whoosh.readthedocs.io/en/latest/

https://pypi.org/project/Whoosh/

lyra https://github.com/nearform/lyra

https://nearform.github.io/lyra/

https://github.com/LyraSearch/lyra

https://lyrasearch.io/

flexsearch

https://github.com/nextapps-de/flexsearch#performance-benchm...

https://pagefind.app/docs/

Lucene

https://github.com/apache/lucene

https://lucene.apache.org/

ZincSearch

https://zincsearch.com/

Solr

https://solr.apache.org/

https://solr.apache.org/operator/

https://solr.apache.org/guide/solr/latest/getting-started/so...

https://github.com/apache/solr

https://solr.apache.org/guide/solr/latest/deployment-guide/s...

Konnu https://gitlab.com/shadowislord/konnu

Quickwit QuickWit + Clickhouse

https://clickhouse.com/docs/en/guides/developer/full-text-se...

https://clickhouse.com/docs/en/sql-reference/functions/strin...

There is no way I can get to running all of these (this project was supposed to be quick!!), but I will run the ones I noted earlier, and probably manticore too since it was high on my list since it's quite polished looking.

Re:An alternative to Elasticsearch that runs on a few MBs of... by kapilvt on Monday October 24, 2022 @12:07PM (#33318533)
+ xapian which has been around a while, and while gpl licensed, is quite capable https://xapian.org/
Re:An alternative to Elasticsearch that runs on a few MBs of... by donio on Monday October 24, 2022 @02:10PM (#33320224)
Xapian is great, especially when you need a a C/C++ library rather than a separate service. Kinda like an sqlite for search. Some of my favorite tools like notmuch and recoll use it.
Re:An alternative to Elasticsearch that runs on a few MBs of... by thirdtrigger on Monday October 24, 2022 @05:05PM (#33322657)
+ Weaviate for vector based search. Has a BSD-3 license. https://weaviate.io/developers/weaviate/current/
Re:An alternative to Elasticsearch that runs on a few MBs of... by fzliu on Monday October 24, 2022 @05:29PM (#33322893)
+ Milvus (https://github.com/milvus-io/milvus) for large scale similarity/semantic search.
Re:An alternative to Elasticsearch that runs on a few MBs of... by _tom_ on Monday October 24, 2022 @06:15PM (#33323361)
I'd encourage you to maintain and publish your list of search engines. Even if you aren't supporting them.

The list has value on its own, especially if you maintain it.

Re:An alternative to Elasticsearch that runs on a few MBs of... by hardwaresofton on Monday October 24, 2022 @08:54PM (#33324781)
Yup this is exactly why I started Awesome F/OSS (https://awsmfoss.com) — I have similar lists for a lot of software and just need to make sure they’re somewhere else other than just my browser!
Re:An alternative to Elasticsearch that runs on a few MBs of... by Bilal_io on Monday October 24, 2022 @10:31AM (#33317140)
Hey that's a great list of tools.

Are you aware of any that can be used client side like Lyra and supports faceted search?

I've been looking for a solution and cannot find it, even an algorithm and/or a data structure can be helpful. I attempted coming up with a solution myself but ended up with frustration when it came to making the facets dynamic and update as other filters are applied.

I read a couple of papers and one stood out [0], which introduces category theory as a solution to faceted filtering. I understood it in theory and it was still does not seem straight forward to implement but I haven't attempted yet.

0. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5145200/#!po=28...

Re:An alternative to Elasticsearch that runs on a few MBs of... by hardwaresofton on Monday October 24, 2022 @10:40AM (#33317294)
So for client-side search, I generally know of Lunr.js:

https://lunrjs.com/docs/index.html

There are some others but I can't find them at this moment -- a bunch of the other projects I find are somewhat abandoned, lunr is actually on my list of things to use (because it makes the most sense to just ship a pre-built index with the first like... 5 letters maybe of typeahead, no matter how fast the backend is)

Re:An alternative to Elasticsearch that runs on a few MBs of... by Bilal_io on Monday October 24, 2022 @11:12AM (#33317778)
Thanks for the link. This unfortunately is not what I am looking for. Faceted filters are a different beast.
Re:An alternative to Elasticsearch that runs on a few MBs of... by francoismassot on Monday October 24, 2022 @11:50AM (#33318301)
You can consider also lnx that is based on tantivy and is performing quite well (https://lnx.rs/).
Re:An alternative to Elasticsearch that runs on a few MBs of... by nightpool on Monday October 24, 2022 @12:45PM (#33319057)
> and if someone is searching on your site excessively maybe there's a problem with your UX (unless you're a search engine or repository of information).

I don't understand this comment. Why would you search something that *isn't*, in some senses, a repository of information? I would say almost every website needs to have search in some sense, and it's *because* sites function as a repository of information that they need this search. Think about e.g. Stripe's documentation, or Github's repository / code search. HN is also another great example—I search for stories or comments all the time to try and remember something I read about recently or heard about last week, but couldn't quite remember. I'm hard-pressed to think of a web site I use regularly that *shouldn't* have full-text search, if I'm being honest.

Re:An alternative to Elasticsearch that runs on a few MBs of... by hardwaresofton on Monday October 24, 2022 @12:53PM (#33319175)
I don't consider use cases like documentation a "repository" of information, but maybe this is just me not phrasing it badly. In the literal sense sure it is, but when I think of a "repository of information" I think of wikipedia, amazon search items, etc.

The scale of a documentation site is a very different problem -- you can brute force it in ways that you can't at larger scales.

I agree that HN would be a case of the large repository, but even then what most people want out of HN search is pretty simple/basic keyword search. I think a decent non-frustrating HN search feature could be very basic and get by without most of the advanced features/rabbit holes available in search.

Basically I think most apps fall into the lighter search use case -- command palettes, search inside of apps with a small scale of information, etc.

My comment wasn't that apps shouldn't have full text search -- it was that most that have full text search don't need complex full text search with all the bells and whistles that lucene and other serious search engines provide. These up-and-comers might be enough for a bunch of apps for which search is not the main feature.

Re:An alternative to Elasticsearch that runs on a few MBs of... by nightpool on Monday October 24, 2022 @07:20PM (#33323943)
I think the sibling comment on this thread kind of disproves this to some extent. Sure, some simple in-memory search library might be good enough for a command palette or for "apps", but the fact that even tiny docs and "marketing" sites that try to implement site search are almost always outdone by google "site:example.com query" really goes to show how much value a full stemming / synonym clustering / syntax normalizing search engine can bring.
Re:An alternative to Elasticsearch that runs on a few MBs of... by TylerE on Monday October 24, 2022 @04:43PM (#33322445)
Most site searches are basically unusable. Either it isn't very good, is painfully slow, or both.

Just gooling site:foo.com/baz <query> almost always produces better results.

Re:An alternative to Elasticsearch that runs on a few MBs of... by MobiusHorizons on Monday October 24, 2022 @03:19PM (#33321158)
Would it make sense to include Sqlite FTS5 in that mix?
Re:An alternative to Elasticsearch that runs on a few MBs of... by hardwaresofton on Monday October 24, 2022 @03:23PM (#33321223)
It would, I did for the supabase post but... This is already way too much! I have no idea when I'll actually be able to get to all this as-is.

Waiting for meilisearch to ingest documents right now and the Show HN is going up.

Re:An alternative to Elasticsearch that runs on a few MBs of... by hardwaresofton on Monday October 24, 2022 @03:35PM (#33321406)
Meili is still ingesting documents but we're live:

https://news.ycombinator.com/item?id=33321268

Maybe I should have used their batch thing instead.

Re:An alternative to Elasticsearch that runs on a few MBs of... by jojo_ on Tuesday October 25, 2022 @01:04AM (#33326448)
Have you heard about Xapian?
Re:An alternative to Elasticsearch that runs on a few MBs of... by hardwaresofton on Tuesday October 25, 2022 @01:09AM (#33326474)
Nope I hadn't, but it was mentioned here:

https://news.ycombinator.com/item?id=33318533