Re:Secor: A service persisting Kafka logs to S3, GCS and Azu...
by onefuncman on Friday March 27, 2020 @11:53AM
(#22703948)
It's not a 1:1 write per message, it relies on Kafka offset guarantees to batch messages into S3 at a configurable threshold, 256MB per S3 object is common.
Re:Secor: A service persisting Kafka logs to S3, GCS and Azu...
by jonathanoliver on Friday March 27, 2020 @06:16PM
(#22707407)
Oh! That's awesome. The way that sentence is written is a little tricky to parse, but the behavior of a batching lots of messages (configurable 256MB+ chunks) is exactly what I'm hoping for.
Re:Secor: A service persisting Kafka logs to S3, GCS and Azu...
by justinsaccount on Friday March 27, 2020 @01:09PM
(#22704677)
What they said:
> it is guaranteed that each message will be saved in exactly one S3 file
What you read:
> it is guaranteed that each S3 file will contain exactly one message
> it is guaranteed that each message will be saved in exactly one S3 file
This can get expensive. Specifically a 1-to-1 write per message with S3 (or any cloud storage provider) gets expensive very, very quickly. Further, when reading from cloud storage, you've got to read one message (object) at a time which adds up as well. I'm not talking about egress bandwidth either, I'm talking about S3 PUT/GET operations. I'd love to see some kind of batching operation that takes groups of messages (perhaps configurable time/size limits?) and writes them as a blog to cloud storage.