News for hackers. Stuff that matters.
Re:An alternative to Elasticsearch that runs on a few MBs of... by PedroBatista on Monday October 24, 2022 @08:10AM (#33315661)
While I get the wants-and-needs since ElasticSearch has a voracious appetite for RAM, I get the feeling most people think search engines are a simple thing where you can just import some lib, fool around for a bit and call it a day.

The truth is that ElasticSearch/Solr/Lucene is orders of magnitude more complex and powerful than these "alternatives". All this is mostly fine as long everyone is on the same page regarding the expectations.

Most people don't need ElasticSearch for their use cases on the surface, but I feel they expect top-notch mind-reading results and that requires something like ElasticSearch and someone who knows the field.

Having said all of that, Meilisearch and this are quite fine.

Re:An alternative to Elasticsearch that runs on a few MBs of... by alessmar on Monday October 24, 2022 @08:30AM (#33315825)
I would like to suggest https://typesense.org/ It has some features that makes it a better choice than Meilisearch
Re:An alternative to Elasticsearch that runs on a few MBs of... by paraboul on Monday October 24, 2022 @10:20AM (#33316954)
Can you elaborate on said features?

I migrated from typesense to Meilisearch on a project after I found it had much better search accuracy. I can't exactly explain why, but overall Meilisearch results feel more relevant by default.

Re:An alternative to Elasticsearch that runs on a few MBs of... by snikolaev on Monday October 24, 2022 @10:39AM (#33317269)
There are actually benchmarks that allow measuring search relevancy objectively, e.g. BEIR[1]. Manticore Search team did an effort to make a PR to include it to the list. The results are here [2]. Unfortunately the BEIR team seems to be too busy to review a whole pile of PRs including about Vespa. Nevertheless it would be nice to have both Meilisearch and Typesense there too since it's interesting what performance those non-tf-idf based search engines would show compared to BM25-based and vector search engines.

[1] https://github.com/beir-cellar/beir [2] https://docs.google.com/spreadsheets/d/1_ZyYkPJ_K0st9FJBrjbZ...

Re:An alternative to Elasticsearch that runs on a few MBs of... by jabo on Monday October 24, 2022 @10:35AM (#33317216)
I work on Typesense. Mind if I ask which version of Typesense and Meilisearch you tried this on? And if this was on some public dataset I can use?

I’d love to take a closer look.

Re:An alternative to Elasticsearch that runs on a few MBs of... by paraboul on Monday October 24, 2022 @12:26PM (#33318786)
Hey jabo,

I migrated in April 2021 (latest version of typesense & meilisearch at that time).

I don't have a public dataset has it was a fairly large ecommerce catalog with close to ~500k entries. And again, it was just my own perception which is hard to define. I just found that Typesense was a bit off compared to Meilisearch on search accuracy, and of course could totally be different today with a more recent release.

Re:An alternative to Elasticsearch that runs on a few MBs of... by jabo on Monday October 24, 2022 @12:47PM (#33319084)
Got it, thank you for sharing that. Typesense was at v0.19.0 around that time. Two prominent issues we had in that version were how we handled matches across multiple fields and how we handled "keyword stuffing".

We're now at v0.24.rc, and we've iterated quite a lot on improving relevancy since then, as more users shared their datasets with us and gave us feedback over the last 1.5 years.

If you get a chance to try out Typesense again in the future, I'd love to hear how relevance feels with the latest version, out of the box for your dataset.

Re:An alternative to Elasticsearch that runs on a few MBs of... by keyle on Monday October 24, 2022 @08:43AM (#33315923)
Yeah there needs to be some kind of acid test that will compare these products on equal footing and show the pitfalls.
Re:An alternative to Elasticsearch that runs on a few MBs of... by sanxiyn on Monday October 24, 2022 @09:02AM (#33316080)
This is very very difficult, but Tantivy tried: see https://github.com/quickwit-oss/search-benchmark-game
Re:An alternative to Elasticsearch that runs on a few MBs of... by jasfi on Monday October 24, 2022 @10:56AM (#33317513)
That would be great. However if you wanted to benchmark relevance ranking, how would you do that?
Re:An alternative to Elasticsearch that runs on a few MBs of... by sanxiyn on Monday October 24, 2022 @11:19AM (#33317853)
You need a dataset and an evaluation metric. The usual evaluation metric is NDCG(Normalized Discounted Cumulative Gain): https://en.wikipedia.org/wiki/NDCG

An example dataset is BEIR(BEnchmarking Information Retrieval), published in NIPS 2021: https://github.com/beir-cellar/beir

Re:An alternative to Elasticsearch that runs on a few MBs of... by DeathArrow on Monday October 24, 2022 @03:47PM (#33321590)
Re:An alternative to Elasticsearch that runs on a few MBs of... by Spivak on Monday October 24, 2022 @12:13PM (#33318613)
I think the upshot is that if you have no idea what all the advanced features of ES even are then you probably don't need ES because it's not turnkey.

If you utter the phrase "I just want search" then it really is a matter of just using one of these lightweight projects and libs because your needs are simple.