Re:Binance built a 100PB log service with Quickwit
by quantified on Thursday July 11, 2024 @09:42PM
(#40942121)
I suspect you and the parent are using different meanings of the word "structured". They're not totally random or they wouldn't be usable. It's a question of what the structuring principle is.
[ Parent
| on HN ]
Re:Binance built a 100PB log service with Quickwit
by KaiserPro on Saturday July 13, 2024 @03:50AM
(#40952526)
How many unique message formats do you think exist in your org?
actually, how many of your messages include the time and date, and how many different ways of displaying timestamp exist in those messages?
That is why I say logs are unstructured, because all but a very few places actually have the discipline to enforce a single log structure.
Re:Binance built a 100PB log service with Quickwit
by charrondev on Saturday July 13, 2024 @09:22AM
(#40953983)
Am I crazy here? We run all of our app logs and error logs through LogStash and just have a few filters in there to normalize stuff like the timestamp. Honestly the only peace of data that absolutely HAS to be standardized because that’s the piece of data that splits our log indexes, is the primary sorting mechanism, and at what point we roll up an index into some aggregates and then compress and send it to cold storage.