News for hackers. Stuff that matters.
Re:Secor: A service persisting Kafka logs to S3, GCS and Azu... by onefuncman on Friday March 27, 2020 @11:53AM (#22703948)
It's not a 1:1 write per message, it relies on Kafka offset guarantees to batch messages into S3 at a configurable threshold, 256MB per S3 object is common.
Re:Secor: A service persisting Kafka logs to S3, GCS and Azu... by jonathanoliver on Friday March 27, 2020 @06:16PM (#22707407)
Oh! That's awesome. The way that sentence is written is a little tricky to parse, but the behavior of a batching lots of messages (configurable 256MB+ chunks) is exactly what I'm hoping for.