News for hackers. Stuff that matters.
Re:Binance built a 100PB log service with Quickwit by endorphine on Thursday July 11, 2024 @09:29AM (#40936589)
What would you use for storing and querying long-term audit logs (e.g. 6 months retention), which should be searchable with subsecond latency and would serve 10k writes per second?

AFAICT this system feels like a decent choice. Alternatives?

Re:Binance built a 100PB log service with Quickwit by bojanz on Thursday July 11, 2024 @09:49AM (#40936816)
You'll find many case studies about using Clickhouse for this purpose.
Re:Binance built a 100PB log service with Quickwit by hipadev23 on Thursday July 11, 2024 @10:03AM (#40936947)
Do you know any specific case studies for unstructured logs on clickhouse?

I think achieving sub-second read latency of adhoc text searching over ~150B rows of unstructured data is going to be quite challenging without a high cost. Clickhouse’s inverted indices are still experimental.

If the data can be organized in a way that is conducive to the searching itself, or structured it into columns, that’s definitely possible. Otherwise I suppose a large number of CPUs (150-300) to split the job and just brute force each search?

Re:Binance built a 100PB log service with Quickwit by buzer on Thursday July 11, 2024 @10:24AM (#40937162)
There is at least https://news.ycombinator.com/item?id=40936947 though it's a bit of mixed in terms how they handle schema.
Re:Binance built a 100PB log service with Quickwit by SSLy on Thursday July 11, 2024 @12:13PM (#40938147)
not sure if an excellent joke or a honest mistake
Re:Binance built a 100PB log service with Quickwit by buzer on Thursday July 11, 2024 @09:46PM (#40942146)
Let's go with former, I definitely didn't mean to link https://www.uber.com/en-FI/blog/logging/ :)
Re:Binance built a 100PB log service with Quickwit by AJSDfljff on Thursday July 11, 2024 @10:10AM (#40937026)
I would question first if the system needs to search with subsecond latency and if the same system needs to be which can handle 10k writes/sec.

Even google cloud and others let you wait for longer search queries. If not business ciritical, you can definitly wait a bit.

And the write system might not need to write it in the endformat. Especially as it also has to handle transformation and filtering.

Nonetheless, as mentioned in my other comment, the interesting details of this is missing.

Re:Binance built a 100PB log service with Quickwit by endorphine on Thursday July 11, 2024 @12:18PM (#40938181)
Let's say that it powers a "search logs" page that an end user wants to see. And let's say that they want last 1d, 14d, 1m, 6m.

So subsecond I would say is a requirement.

And no, it doesn't have to be the same system that ingests/indexes the logs.

Re:Binance built a 100PB log service with Quickwit by AJSDfljff on Friday July 12, 2024 @04:56AM (#40943839)
"So subsecond I would say is a requirement." you do not make any specific point why you came to that conclusion.

You can easily entertain users to show them that the system is doing something in the background without loosing them and if they are collegues who actually need to search, you don't even need to keep them as they have to use your setup.

Re:Binance built a 100PB log service with Quickwit by endorphine on Friday July 12, 2024 @09:34AM (#40945427)
OK, let's say it needs to be <3s, for reasons.
Re:Binance built a 100PB log service with Quickwit by SSLy on Thursday July 11, 2024 @10:07AM (#40936988)
What if I don't have such latency requirements? I'm willing to trade that for flexibility or anything else
Re:Binance built a 100PB log service with Quickwit by jjordan on Thursday July 11, 2024 @09:47AM (#40936788)
NATS?
Re:Binance built a 100PB log service with Quickwit by packetlost on Thursday July 11, 2024 @09:51AM (#40936832)
NATS doesn't really have advanced query features though. It has a lot of really nice things, but advanced querying isn't one of them. Not to mention I don't know if NATS does well with large datasets, does it have sharding capability for it's KV and object stores?
Re:Binance built a 100PB log service with Quickwit by Zambyte on Thursday July 11, 2024 @10:45AM (#40937354)
I use NATS at work, and I have had the privilege to speak with some of the folks at Synadia about this stuff.

Re: advanced querying: the recommended way to do this is to build an index out of band (like Redis (or a fork) or SQLite or something) that references the stored messages by sequence number. By doing that, your index is just this ephemeral thing that can be dynamically built to exactly optimize for the queries you're using it for.

Re: sharding: no, it doesn't support simple sharding. You can achieve sharding by standing up multiple NATS instances, and making a new stream (KV and object store are also just streams) on each instance, and capture some subset of the stream on each instance. The client (or perhaps a service querying on behalf of the client) would have to me smart enough to be able to mux the sources together.

Re:Binance built a 100PB log service with Quickwit by packetlost on Thursday July 11, 2024 @03:41PM (#40939865)
Does it handle clustering/redundancy for the data stored in KV/object store? My intuition says yes because I believe it supports it at the "node" level
Re:Binance built a 100PB log service with Quickwit by Zambyte on Thursday July 11, 2024 @05:33PM (#40940804)
Yes. When you create a stream (including a KV or object store) you say what cluster you want to put it on, and how many replicas you want it to have.
Re:Binance built a 100PB log service with Quickwit by packetlost on Thursday July 11, 2024 @09:15PM (#40941992)
Very cool, I'll have to keep that in mind next time I'm in need of something similar!
Re:Binance built a 100PB log service with Quickwit by jakjak123 on Thursday July 11, 2024 @04:45PM (#40940470)
10k audit logs per sec? I think we have different definitions of audit logs.