Genuine question - Why does everyone seem to think running a zookeeper cluster is so hard? You can run it on three small VMs and basically forget about it. We didn't have any zookeeper experience at my last startup before we started using it for Kafka and we used a very simple puppet module to install it on three instances in each of our AWS regions. It really never gave us many problems in the several years since.

Also, all the tooling around it is quite mature - there are great monitoring and management tools for probing at the internals which helped when we were dealing with more exotic kafka surgery.

For some reason Zookeeper is unjustly seem as uncool technology. I even seen it being blamed for issues that it had nothing to do with.

People say that setting ZK cluster is a huge issue, yet they don't see a problem spinning etcd, or sentry nodes in case of redis.

When I learned about ZK I was skeptic, didn't like that it was written in Java, but ZK proved to be extremely robust.

I'm not sure about it being "so hard", but it's extra stuff to deploy, maintain, monitor, and pay for. Very small teams benefit from keeping things small and simple. Again, if you actually need Kafka, then it's worth it. If you just need something similar, but can deal with the limitations of redis streams, then it's an easy choice.

Because nothing in the current hype cycle depends on zookeeper so people use that as a crutch for following hype.