Re:Redis Streams and the Unified Log
by netcraft on Wednesday November 08, 2017 @03:30PM
(#15656653)
Is Kafka expensive because of the hardware requirements, the number of servers needed or just because its relatively niche? Or something else?
[ Parent
| on HN ]
Re:Redis Streams and the Unified Log
by WaxProlix on Wednesday November 08, 2017 @03:35PM
(#15656699)
For simpler setups, you can just search for "kafka" in the AWS marketplace and snag a bitnami instance. Larger clusters take more administrative overhead; I've had to set a few of these up manually and it's not fun or particularly intuitive.
Re:Redis Streams and the Unified Log
by tomnipotent on Wednesday November 08, 2017 @03:35PM
(#15656700)
Common complaint is the reliance on ZooKeeper, which requires three servers to get started.
Kafka requires only a single ZooKeeper instance to get started. Kafka's official quickstart actually walks you through running a local deployment of Kafka that has 1x Kafka broker and 1x ZK instance: https://kafka.apache.org/documentation/#quickstart
Re:Redis Streams and the Unified Log
by WaxProlix on Wednesday November 08, 2017 @03:40PM
(#15656750)
Yeah and it's not obvious that there's any way to get Kafka up and running without Zookeeper, or what their interaction is. And you can get Zookeeper and Kafka running on one machine, but it's such a pain and there's nothing to help really in the docs.
Re:Redis Streams and the Unified Log
by poooogles on Wednesday November 08, 2017 @06:10PM
(#15658034)
>And you can get Zookeeper and Kafka running on one machine, but it's such a pain and there's nothing to help really in the docs.
What sort of thing are you after? Like a getting started for production without creating footguns style how too?
Re:Redis Streams and the Unified Log
by WaxProlix on Wednesday November 08, 2017 @10:11PM
(#15659514)
More like getting started in a dev environment; I want to play with a technology on a laptop or my home compute machine sometimes before venturing into production or recommending it to clients.
Also, since you talked above about the "lack of documentation" to get up and running with a small Kafka deployment of only 1 broker and 1 ZK instance: this is covered in the Kafka quickstart, which is probably what most people will read when starting out with Kafka. See https://kafka.apache.org/documentation/#quickstart.
Perhaps you had a different issue though?
Why don't you run Kafka from a Docker image, like the Docker images provided by Confluent?https://docs.confluent.io/current/installation/docker/docs/