News for hackers. Stuff that matters.
Re:Secor: A service persisting Kafka logs to S3, GCS and Azu... by jonathanoliver on Friday March 27, 2020 @11:51AM (#22703930)
I love this. This is a property already baked into another messaging queuing system called Pulsar and I've thought about writing my own for internal use (maybe to be open-sourced). The only quirk that I see is this link in the project description readme:

> it is guaranteed that each message will be saved in exactly one S3 file

This can get expensive. Specifically a 1-to-1 write per message with S3 (or any cloud storage provider) gets expensive very, very quickly. Further, when reading from cloud storage, you've got to read one message (object) at a time which adds up as well. I'm not talking about egress bandwidth either, I'm talking about S3 PUT/GET operations. I'd love to see some kind of batching operation that takes groups of messages (perhaps configurable time/size limits?) and writes them as a blog to cloud storage.

Re:Secor: A service persisting Kafka logs to S3, GCS and Azu... by onefuncman on Friday March 27, 2020 @11:53AM (#22703948)
It's not a 1:1 write per message, it relies on Kafka offset guarantees to batch messages into S3 at a configurable threshold, 256MB per S3 object is common.
Re:Secor: A service persisting Kafka logs to S3, GCS and Azu... by jonathanoliver on Friday March 27, 2020 @06:16PM (#22707407)
Oh! That's awesome. The way that sentence is written is a little tricky to parse, but the behavior of a batching lots of messages (configurable 256MB+ chunks) is exactly what I'm hoping for.
Re:Secor: A service persisting Kafka logs to S3, GCS and Azu... by justinsaccount on Friday March 27, 2020 @01:09PM (#22704677)
What they said:

> it is guaranteed that each message will be saved in exactly one S3 file

What you read:

> it is guaranteed that each S3 file will contain exactly one message

Re:Secor: A service persisting Kafka logs to S3, GCS and Azu... by otterley on Friday March 27, 2020 @01:17PM (#22704759)
I think we can all agree that the verbiage could be more elucidating and less confusing.