Re:Streaming Databases in Real-Time with Kafka, Debezium, an...
by dantiberian on Thursday March 30, 2017 @07:08PM
(#13999728)
> Because Kafka message expiry has only two policies, as far as I could tell: time or space.
You can use log compaction to remove all messages apart from the latest for any given key. This gives you a Kafka queue with a bounded size, that is proportional to the table size that you're replicating.
You can use log compaction to remove all messages apart from the latest for any given key. This gives you a Kafka queue with a bounded size, that is proportional to the table size that you're replicating.
https://kafka.apache.org/documentation/#compaction