Re:Apache Kafka Goes 1.0
by breischl on Wednesday November 01, 2017 @03:33PM
(#15604742)
IDK about Kinesis, but for Kafka there are two major cases here:
- If your Kafka retention period is long enough, then the data will still be sitting in Kafka. You don't need to replay, just have the new consumer start from the beginning.
- If the retention period is not long enough, then you'll have to replay data. There's not any provision for that in Kafka itself - it's really just a somewhat-smart logfile. But whatever you're using to write into Kafka may have some way to do it. This is similar to most message queuing or eventing systems. eg, RabbitMQ.
- If your Kafka retention period is long enough, then the data will still be sitting in Kafka. You don't need to replay, just have the new consumer start from the beginning.
- If the retention period is not long enough, then you'll have to replay data. There's not any provision for that in Kafka itself - it's really just a somewhat-smart logfile. But whatever you're using to write into Kafka may have some way to do it. This is similar to most message queuing or eventing systems. eg, RabbitMQ.