Re:Pulsar – an open-source distributed pub-sub messaging pla...
by imglorp on Monday July 13, 2020 @06:37AM
(#23819253)
Hasn't Kafka been talking about eliminating external zk forever? Are they making it an internal library or adopting a new consensus scheme?
Re:Pulsar – an open-source distributed pub-sub messaging pla...
by miguno on Monday July 13, 2020 @07:07AM
(#23819427)
Yes, you remember correctly. The Kafka community has already started the work on getting rid of ZooKeeper. See https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A... for details.
Re:Pulsar – an open-source distributed pub-sub messaging pla...
by FridgeSeal on Monday July 13, 2020 @07:14AM
(#23819486)
From my understanding, they’re going to use the brokers as their own metadata store, so the data that was in ZK goes in a topic instead, and I believe Raft for leadership?
KIP-500 is the proposal/PR (?) for ZK-free Kafka which I understand has been merged in and work is now progressing on follow up tasks.
I would argue that building up on existing systems (especially things like a distributed consensus system like Zookeeper) is generally good practice. Being able to just point a bunch of self-contained JARs/WARs at a Zookeeper/etcd cluster and not worry about startup sequencing or cluster bootstrapping makes me happy.