Re:Binance built a 100PB log service with Quickwit
by fulmicoton on Thursday July 11, 2024 @09:51AM
(#40936834)
Quickwit is designed to do full-text search efficiently with an index stored on an object storage.
There are no equivalent technology, apart maybe:
- Chaossearch but it is hard to tell because they are not opensource and do not share their internals. (if someone from chaossearch wants to comment?)
- Elasticsearch makes it possible to search into an index archived on S3. This is still a super useful feature as a way to search punctually into your archived data, but it would be too slow and too expensive (it generates a lot of GET requests) to use as your everyday "main" log search index.
Re:Binance built a 100PB log service with Quickwit
by BiteCode_dev on Thursday July 11, 2024 @09:58AM
(#40936912)
Click house does have it, but it's experimental.
Maybe Quickwit is that indexing layer in this case? I haven't dug too much into the general state of cloud dw indexing.