News for hackers. Stuff that matters.
Re:Binance built a 100PB log service with Quickwit by hipadev23 on Thursday July 11, 2024 @10:03AM (#40936947)
Do you know any specific case studies for unstructured logs on clickhouse?

I think achieving sub-second read latency of adhoc text searching over ~150B rows of unstructured data is going to be quite challenging without a high cost. Clickhouse’s inverted indices are still experimental.

If the data can be organized in a way that is conducive to the searching itself, or structured it into columns, that’s definitely possible. Otherwise I suppose a large number of CPUs (150-300) to split the job and just brute force each search?

Re:Binance built a 100PB log service with Quickwit by buzer on Thursday July 11, 2024 @10:24AM (#40937162)
There is at least https://news.ycombinator.com/item?id=40936947 though it's a bit of mixed in terms how they handle schema.
Re:Binance built a 100PB log service with Quickwit by SSLy on Thursday July 11, 2024 @12:13PM (#40938147)
not sure if an excellent joke or a honest mistake
Re:Binance built a 100PB log service with Quickwit by buzer on Thursday July 11, 2024 @09:46PM (#40942146)
Let's go with former, I definitely didn't mean to link https://www.uber.com/en-FI/blog/logging/ :)