PSeitz

Hi, tantivy dev here. There are two recent performance improvements in tantivy, which should make term aggregations considerable faster.

https://github.com/quickwit-oss/tantivy/pull/2740 https://github.com/quickwit-oss/tantivy/pull/2759

Also we are working on some performance improvements based on the benchmark comparison, as they highlighted some areas we can improve in tantivy.

tantivy has two dictionaries FST and SSTable. We added SSTable in tantivy because it works great with object storage, while FST does not. With some metadata we can download only the required parts and not the whole dictionary.

SStable does not support Regex queries, it would require a full load and scan, which would be very expensive.

Your best bet currently would be to make it work with tokenizing, which is way more efficient anyways.

prefix queries are supported btw

You can just put everything in a JSON field in tantivy and set it to INDEXED and FAST

They serve quite different use cases.

quickwit was built to handle extremely large data volumes, you can ingest and search TB and PB of logs.

meilisearches indexing doesn't scale as it will become slower the more data you have, e.g. I failed to ingest 7GB of data.

> Hm, I am interested, but I would love to use it as a rust lib and just have rust types instead of some json config...

Yes that's how you use tantivy normally, not sure which json config you mean.

tantivy-cli is more like a showcase, https://github.com/quickwit-oss/tantivy is the actual project.

The issue for geo search is here: https://github.com/quickwit-oss/tantivy/issues/44

>Roaring bitmaps and similar data structures get their speed from decoding together consecutive groups of elements, so if you do sequential decoding or decode a large fraction of the list you get excellent performance. EF instead excels at random skipping, so if you visit a small fraction of the list you generally get better performance. This is why it works so well for inverted indexes, as generally the queries are very selective (otherwise why do you need an index?) and if you have good intersection algorithms you can skip a large fraction of documents.

There's no sequential decoding in our variant, every access is independent. The roaring bitmap variant is used only for the optional index (1 docid => 0 or 1 value) in the columnar storage (DocValues), not for the inverted index. Since this is used for aggregation, some queries may be a full scan.

The inverted index in tantivy uses bitpacked values of 128 elements with a skip index on top.

> I didn't follow the rest of your comment, select is what EF is good at, every other data structure needs a lot more scanning once you land on the right chunk. With BMI2 you can also use the PDEP instruction to accelerate the final select on a 64-bit block

The select for the sparse codec is a simple array index access, that is hard to beat (https://github.com/quickwit-oss/tantivy/blob/main/columnar/s...). Compression is not good near the 5k threshold though. PDEP is currently deactivated since ryzen before Zen3 was really slow.

Creation speed is also quite important, do you know how "Partitioned Elias-Fano" performs there?

> It would definitely compress much better than roaring bitmaps. In terms of performance, it depends on the access patterns. If very sparse (large jumps), PEF would likely be faster, if dense (visit a large fraction of the bitmap), it'd be slower.

Just for clarification you mean the access pattern is sparse and not the data itself? How is that relevant?

I did part of the investigation and implementation, but didn't look much into elias fano coding. The select for the dense codec is already really fast (with popcount), there's not much room for improvement on the instruction side (https://godbolt.org/z/dq7WeE66Y), except implicitly by touching less memory. The sparse codec with its binary search should be easy to beat though. Partitioned Elias-Fano indexes may be a superior choice in contrast to the sparse codec in terms of rank and compression, and probably less so for select and code complexity.

Indeed if you are colorblind that may be an issue. I'll address it If I get to it. Otherwise it assigns names to colors, no idea how you could be confused by that.

file io should come with the frame format, which is not yet implemented. For the block format it's not really suited.

As far as I know LZ4 is much faster that most compression algorithms, with decompression speeds of over 4GB/s

I ported the block format to Rust matching the C implementation in performance and ratio.

https://github.com/pseitz/lz4_flex

But I like coal, so coal can't be the problem either. ROLF!

Or with einstein "yeah you were pretty much right. we don't have anything new"

The performance comparisons between gc and non-gc languages show otherwise. Also, memory consumption eventually costs cpu cycles.

Garbage collector is actually something you'd want, if it caused no performance hit.

Some article about development a approach is not news. Also, hackernews is largely composed of non-news information, else I wouldn't be here ;)

I use a state machine for text

https://github.com/BurntSushi/fst

But it does work in Tokyo, maybe it's the people

Internet availability is not a problem. Just check cable availability and your up and running with 100Mbits+ in a few days.

And Germans really like to complain.

If your editor is your compiler, then something is seriously wrong.

you always need at least compiler/interpreter to execute your code... unless write bytecode in your editor

Seems to me, that apple bosses want to put their name on every patent created in their division.

I heard that about Atom around 12 months ago.