Re:Binance built a 100PB log service with Quickwit
by endorphine on Thursday July 11, 2024 @12:18PM
(#40938181)
Let's say that it powers a "search logs" page that an end user wants to see. And let's say that they want last 1d, 14d, 1m, 6m.
So subsecond I would say is a requirement.
And no, it doesn't have to be the same system that ingests/indexes the logs.
Re:Binance built a 100PB log service with Quickwit
by AJSDfljff on Friday July 12, 2024 @04:56AM
(#40943839)
"So subsecond I would say is a requirement." you do not make any specific point why you came to that conclusion.
You can easily entertain users to show them that the system is doing something in the background without loosing them and if they are collegues who actually need to search, you don't even need to keep them as they have to use your setup.
Re:Binance built a 100PB log service with Quickwit
by endorphine on Friday July 12, 2024 @09:34AM
(#40945427)
OK, let's say it needs to be <3s, for reasons.
Even google cloud and others let you wait for longer search queries. If not business ciritical, you can definitly wait a bit.
And the write system might not need to write it in the endformat. Especially as it also has to handle transformation and filtering.
Nonetheless, as mentioned in my other comment, the interesting details of this is missing.