News for hackers. Stuff that matters.
Re:Apache Kafka Goes 1.0 by manigandham on Wednesday November 01, 2017 @03:37PM (#15604773)
There's nothing special about replay when it comes to a log. The only issue is retention and many managed services have limited retention based on data size or time frame. S3 archiving is common to keep the operational dataset small so reading it back and refilling Kafka for another consumer is actually pretty normal.

If you run Kafka yourself, you can set an infinite retention and keep all the messages, and also use the compaction feature to keep the latest value for every key to reduce some of the bloat if that applies to your data, but if you don't need it instantly accessible then it's probably worth archiving to cheaper storage.

There are no comments on this comment yet.