wolfgarbe

Last time I posted SeekStorm here, it was lexical-only. Since then, SeekStorm has gained native vector and hybrid search, a sharded, multi-core index architecture (think of a miniature data center inside one machine), faster phrase search with an N-gram index, multiple document compression schemes, result sorting, typo-tolerant search and autocomplete, PDF ingestion, SDKs for multiple languages, and now an MCP server.

A lot has changed in two years!

Peter Norvig shows that an edit distance = 2 will cover 98.9% spelling errors. https://impythonist.wordpress.com/2014/03/18/peter-norvigs-2...

That's the reason why the default maximum edit distance of SymSpell is 2.

Now, all your 6 out of 6 examples are chosen from that 1.1 % margin that is not covered by edit distance 2, presenting a rather unlikely high amount of errors within a single word.

The third-party SymSpell port from Justin Willaby, which you were using for benchmarking, clearly states that you need to set both maxEditDistance and dictionaryEditDistance to a higher number if you want to correct higher edit distances. That you neither used nor mentioned. This has nothing to do with accuracy; it is a performance vs. maximum edit tradeoff one can make according to the use case at hand.

https://github.com/justinwilaby/spellchecker-wasm?tab=readme...

pronnouncaition IS within edit distance 3, according to the Damerau-Levenshtein edit distance used by SymSpell. The reason is that adjacent transpositions are counted as a single dit. https://en.wikipedia.org/wiki/Damerau%E2%80%93Levenshtein_di...

Author of SymSpell here. Congrats on the launch of Lexiathan.

Unfortunately, the comparison of Lexiathan vs. Symspell on your website regarding accuracy is misleading.

1. SymSpell has two parameters to control the maximum edit distance. Once you set both to 3, then also terms with an edit distance of 3 are accurately corrected:

  pronnouncaition -> pronunciation

  inndappendent -> independent

  unegspeccted -> unexpected

  soggtwaee       -> software

2. SymSpell comes with dictionaries in several sizes. Once you load the 500_000 terms dictionary, then also the two remaining terms will be corrected:

  maggnificntally -> magnificently

  annnesteasialgist -> anesthesiologist

https://github.com/wolfgarbe/SymSpell/blob/master/SymSpell.B...

Because SymSpell is Open-Source, everyone can integrate it into their applications for free, modify the code, use different dictionaries in various languages, or add terms to existing ones.

https://github.com/wolfgarbe/SymSpell

https://github.com/wolfgarbe/symspell_rs

The stopword list in SeekStorm is purely optional, per default it is empty.

The query "to be or not to be" that you mentioned, consisting solely of stopwords, returns complete results and perform quite well in the benchmark: https://github.com/SeekStorm/SeekStorm?tab=readme-ov-file#be...

Both Lucene and Elastic still offer stopword filters: https://lucene.apache.org/core/10_3_2/analysis/common/org/ap... https://www.elastic.co/docs/reference/text-analysis/analysis...

Can the index size exceed the RAM size (e.g., via memory mapping), or are index size and document number limited by RAM size? It would be good to mention those limitations in the README.

Sure, but it says "High-performance" Full Text Search Engine. Shouldn't that claim be backed up by numbers, comparing it to the state of the art?

Great work! Would be interesting to see how it compares to Lucene performance-wise, e.g. with a benchmark like https://github.com/quickwit-oss/search-benchmark-game

The most widely used DHT is Kademlia from Petar Maymounkov and David Mazières. It is used in Ethereum, IPFS, I2P, Gnutella DHT, and many other applications.

https://en.wikipedia.org/wiki/Kademlia

https://pdos.csail.mit.edu/~petar/papers/maymounkov-kademlia...

https://web.archive.org/web/20120128120732/http://www.cs.ric...

SeekStorm does currently not use io_uring, but it is on our roadmap. Challenges are the cross-platform compatibility. Linux (io_uring) and Windows (IoRing) use different implementations, and other OS don't support it. There is no abstraction layer over those implementations in Rust, so we are on our own.

It would increase concurrent read and write speed (index loading, searching) by removing the need to lock seek and read/write.

But I would expect that the mmap implementations do already use io_uring / IoRing.

Yes, lazy loading would be possible, but pure RAM access does not offer enough benefits to justify the effort to replicate much of the memory mapping.

SeekStorm comes with an http interface.

The SeekStorm server features an REST API via http: https://seekstorm.apidocumentation.com

It also comes with an embedded Web UI: https://github.com/SeekStorm/SeekStorm?tab=readme-ov-file#bu...

Or did you mean a Web based interface to create and manage indices, define index schemas, add documents etc?

>> The documentation seems a bit sparse.

We just released a new OpenAPI based documentation for the SeekStorm server REST API: https://seekstorm.apidocumentation.com

For the library we have the standard rust doc: https://docs.rs/seekstorm/latest/seekstorm/

For the latency benchmarks we used vanilla BM25 (SimilarityType::Bm25f for a single field) for comparability, so there are no differences in terms of accuracy.

For SimilarityType::Bm25fProximity which takes into account the proximity between query term matches within the document, we have so far only anecdotal evidence that it returns significantly more relevant results for many queries.

Systematic relevancy benchmarks like BeIR, MS MARCO are planned.

The Seekstorm library is 9 MB, and the Seekstorm server executable is 8 MB, depending on the features selected in cargo.

You add the library via 'cargo add seekstorm' to your project which you anyway have to compile.

As for the server, we may add binaries for the main OS in the future.

WASM and Python bindings are on our roadmap.

In SeekStorm you can choose per index whether to use Mmap or let SeekStorm fully control Ram access. There is a slight performance advantage to the latter, at the cost of higher index load time of the former. https://docs.rs/seekstorm/latest/seekstorm/index/enum.Access...

The benchmark should be fairly fair, as it was developed by Tantivy themselves (and Jason Wolfe). So, the choice of corpus and queries was theirs. But, of course, your mileage may vary. It is always best to benchmark it on your machine with your data and your queries.

Yes, WASM and Python bindings are on our roadmap.

Currently, you can choose between tokenizers with or without folding. But configurability per language or full customizability of the folding logic by the user is a good idea.

The code for the distributed search cluster is not yet stable enough to be published, but it will be released as open-source as well.

As for shared storage, do you mean something like NAS or, rather Amazon S3? Cloud-native support of object storage and separating storage and compute is on our roadmap. Challenges will be maintaining latency and the need for more sophisticated caching.

Its not just about speed. Speed reflects efficiency. Efficiency is needed to serve more queries in parallel, to search within exponentially growing data, with less expensive hardware, and fewer servers, consuming less energy. Therefore the pursuit for efficiency never gets outdated and has no limit.

PostgreSQL is an SQL database that also offers full-text search (FTS), with extensions like pg_search it also supports BM25 scoring which is essential for lexical search. SeekStorm is centered around full-text search only, it doesn't offer SQL.

Performance-wise it would be indeed interesting to run a benchmark. The third-party open-source benchmark we are currently using (search_benchmark_game) does not yet support PostgreSQL. So yes, that comparison is still pending.

Yes. We waited long for AOT compilation to become mature, to remove the need for the user to install the .Net framework. But two years ago when we decided to switch, we still couldn't just get the AOT compilation of our codebase to work without changes (perhaps it was somehow possible, but the available documentation was not very verbose about this). Also, there is still a performance gap. Of course, this doesn't matter for most of the applications, where the completeness and consistency of the framework, and the number of programmers fluent in that language might matter more. But for a search server, we needed to carve out every inch of performance we could get. And other benchmarks seemed to echo our experience: https://programming-language-benchmarks.vercel.app/rust-vs-c...

The 2-4 speed ratio was not meant to denounce C#, which is a great language I loved to program in for over two decades, coming from Delphi. Unfortunately, C# has not a complete SIMD support. See our request to support the SSE4.2 _mm_cmpistrm instruction https://github.com/dotnet/runtime/discussions/63332, which we required for a vectorized intersection between two sorted 16-bit arrays. We did the switch from C# to Rust not light-minded, as the cost of porting a fairly large codebase is time-consuming. We just wanted to share our experience for our specific task, not as a general statement.

We started with making the core search technology faster. Then we added a Unicode character folding/normalization tokenizer (diacritics, accents, umlauts, bold, italic, full-width chars...). Last week we added a tokenizer that supports Chinese word segmentation. Currently, we are working on a multi-language tokenizer, that segments Chinese, Japanese an Korean without switching the tokenizer.

Yes, integration in complex legacy systems is always challenging. As a small startup, we are concentrating on core search technology to make search faster and to make the most of available server infrastructure. As SeekStorm is open-source, system integrators can take it from there.

It depends on the application.

When using SeekStorm as a server, keeping the latency per query low increases the throughput and the number of parallel queries a server can handle on top of a given hardware. An efficient search server can reduce the required investments in server hardware.

In other cases, only the local search performance matters, e.g., for data mining or RAG.

Also, it's not only about averages but also about tail latencies. While network latencies dominate the average search time, that is not the case for tail latencies, which in turn heavily influence user satisfaction and revenue in online shopping.

Series of blog posts on search and information retrieval

https://medium.com/@dtunkelang

What AI Engineers Should Know about Search from Doug Turnbull

https://softwaredoug.com/blog/2024/06/25/what-ai-engineers-n...

Videos from Berlin Buzzwords

https://www.youtube.com/@PlainSchwarzUG/videos

Videos MIx-Camp E-commerce Search

https://www.youtube.com/@mix-campe-commercesearch2961/videos

Videos from OpenSource Connections / Haystack

https://www.youtube.com/@OpenSourceConnections/videos

This is no contradiction, you can concentrate on the algorithm also in a faster language :-) https://benchmarksgame-team.pages.debian.net/benchmarksgame/... https://benchmarksgame-team.pages.debian.net/benchmarksgame/... https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

A laudable effort. Two questions:

1. What is the rationale behind choosing Python as a implementation language? Performance and efficiency are paramount in keeping operational costs low and ensuring a good user experience even if the search engine will be used by many users. I guess Python is not the best choice for this, compared to C, Rust or Java.

2. What is the rationale behind implementing a search engine from scratch versus using existing Open Source search engine libraries like Apache Lucene, Apache Solr and Apache Nutch (crawler)?

If you click a tag on the result/preview page (in mobile Firefox), the result list is filtered by the tag (you get a new/shorter result list with an updated result number).

But because you are still on the preview page you don't immediately see this as it's done in the background. But once you close the preview page (with the big arrow on the top/left of the preview page) you will see the updated result list and number.

Of course, this is a design flaw, we should automatically close the preview window once you select a tag to immediately show you the updated result list. In the desktop version, this is no issue, as there result list and preview window are always simultaneously visible.