News for hackers. Stuff that matters.
Re:Binance built a 100PB log service with Quickwit by gaogao on Thursday July 11, 2024 @09:35AM (#40936662)
Yeah, doing some preferred cloud Data Warehouse with an indexing layer seems fine for this sort of thing. That has an advantage over something specialized like this of still being able to easily do stream processing / Spark / etc, plus probably saves some money.

Maybe Quickwit is that indexing layer in this case? I haven't dug too much into the general state of cloud dw indexing.

Re:Binance built a 100PB log service with Quickwit by fulmicoton on Thursday July 11, 2024 @09:51AM (#40936834)
Quickwit is designed to do full-text search efficiently with an index stored on an object storage.

There are no equivalent technology, apart maybe:

- Chaossearch but it is hard to tell because they are not opensource and do not share their internals. (if someone from chaossearch wants to comment?)

- Elasticsearch makes it possible to search into an index archived on S3. This is still a super useful feature as a way to search punctually into your archived data, but it would be too slow and too expensive (it generates a lot of GET requests) to use as your everyday "main" log search index.

Re:Binance built a 100PB log service with Quickwit by BiteCode_dev on Thursday July 11, 2024 @09:58AM (#40936912)
Click house does have it, but it's experimental.