Publishing with Apache Kafka at The New York Times

Re:Publishing with Apache Kafka at The New York Times by pram on Wednesday September 06, 2017 @06:07PM (#15187524)

Kafka by default is not persistent, the logs expire after 7 days. You can increase it on a per topic basis. It also doesn't scale automatically. If you have three replicas on a single partition topic, they will live on their assigned broker forever unless you manually reassign them. Adding new nodes does not kick off rebalancing of partitions. Its automatic cluster management is very primitive compared to something like elasticsearch.

For example, if you lose a broker, the replica will just be gone forever. Unless you replace the broker with the same id.

[ Parent | on HN ]

Re:Publishing with Apache Kafka at The New York Times by tlberglund on Wednesday September 06, 2017 @10:47PM (#15188812)

Your statements about data retention are true, but if I may, I think they are only vacuously true. There are definitely ways to configure data retention incorrectly, but it seems like the folks at NYT have cracked the nut. Indeed, this is a trivial thing to set up.

Your points about auto-scaling are well-taken, though. It does not automatically rebalance when you add a new node. Confluent obvious agrees that this is a problem, since this is part of what Confluent Enterprise does. #shamelessplug (Which of course I do not intend shamelessly; just saying you have a point. :) )

[ Parent | on HN ]