Re:Kafka without ZooKeeper
by sumtechguy on Wednesday March 31, 2021 @09:05AM
(#26646729)
Oh it most certainly simplifies things. I am looking at half the number of boxes needed to run. Which is not insignificant in my cost structure.
What is the migration strategy here? Is it doc'd up yet? I am having flashbacks to migration for follower partitions recently which required a decent amount of pre planning of partition layout.
Also as it is pulling in the duties of ZK into kafka what sort of CPU/memory changes are you seeing? Is it 'meh' or all the way to 'you may want to add a couple of CPUs and a few more GB'? Also is it working ok with the stretched cluster?
Also if you want to hit an interesting market you may want to look at 'does it run OK on a raspberry PI'.
A nit regarding the disclosure: I prefer it at the top of the message, and I think that's "best practice", but I don't know for sure.
Your clarification made me wonder:
Is the single process deployment only doable via a container? Or will we actually have OS native process as well?
Re:Kafka without ZooKeeper
by colin_mccabe on Tuesday March 30, 2021 @10:12PM
(#26642800)
Yes, you can run a single OS native process in KRaft mode, without using Docker. Docker just avoids the need to install a JVM, but it is not required.
Going forward, you will no longer need to configure and run a separate ZooKeeper service just to run Kafka. For proof-of-concept projects, a single-process Docker image will be available when running in KRaft mode (non-ZK mode).
For bigger projects, you may want to use a managed cloud service. Or if you do choose to manage it yourself, it will be easier running one service than two.
Disclosure: I work for Confluent.