KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

Re:KIP-500: Replace ZooKeeper with a Self-Managed Metadata Q... by czinck on Friday August 02, 2019 @03:55PM (#20596063)

Reading the proposal, it seems like they want to replace it because the ZooKeeper API is causing some issues for them, so even if ZooKeeper has perfect consensus, Kafka itself won't necessarily. From the "Metadata as an Event Log" section

> although ZooKeeper is the store of record, the state in ZooKeeper often doesn't match the state that is held in memory in the controller

Maybe it would be better to put the effort in to changing ZooKeeper instead of writing their own consensus system in Kafka, but I would assume they know that tradeoff better than us since they work so closely with ZooKeeper.

[ Parent | on HN ]

Re:KIP-500: Replace ZooKeeper with a Self-Managed Metadata Q... by rad_gruchalski on Friday August 02, 2019 @04:00PM (#20596104)

I do find that statement really confusing. That implies they could have errors in the code handling the consensus? Hard to believe but otoh, they’re not really saying anything about zookeeper causing problems.

Maybe the correct solution is to understand why the states don’t match in the first place?

[ Parent | on HN ]