News for hackers. Stuff that matters.
Re:Apache Kafka Goes 1.0 by slap_shot on Wednesday November 01, 2017 @03:11PM (#15604542)
Interesting. Can someone give a quick/high level list what makes it better than Kafka and in what use cases?
Re:Apache Kafka Goes 1.0 by manigandham on Wednesday November 01, 2017 @03:50PM (#15604853)
I recommend reading the Pulsar architecture page: https://pulsar.apache.org/docs/latest/getting-started/Concep...

Biggest difference is that Pulsar splits compute from storage. Pulsar brokers are stateless services that manage topics and handle clients (publishers and consumers) while reading/writing data from a lower tier of bookies, which is the Apache BookKeeper project for low-latency real-time storage workloads.

Better performance and easier scaling compared to Kafka's partition rebalancing. It also natively supports data center replication and multiple clusters working together globally. Pulsar addressing is property/cluster/namespace/topic so multi-tenant isolation is built-in.

Anywhere you use Kafka can use Pulsar and also consolidate pub/sub only systems that need lower latency than Kafka.

Re:Apache Kafka Goes 1.0 by lmsp on Wednesday November 01, 2017 @09:07PM (#15607012)
I think it is beyond partition rebalancing. There is a fact that people didn't realize of making message broker `stateless`. It is actually much better on reacting to failures or shifting traffic, which is critical when running a messaging bus for online services. because it doesn't have to wait for copying the data of a whole partition when error occurs.
Re:Apache Kafka Goes 1.0 by manigandham on Wednesday November 01, 2017 @11:04PM (#15607515)
Agreed, the architecture page and other blog posts do a more thorough job of explaining the details. Having a stateless broker layer on top of a focused data layer makes all the operations much easier. I expect BookKeeper to further integrate with the various cloud storage APIs so it can also start to become stateless cache.
Re:Apache Kafka Goes 1.0 by majidazimi on Wednesday November 01, 2017 @04:34PM (#15605217)
The biggest differences:

- rebalancing: Bookkeeper (which is the storage backend of Pulsar) manages rebalancing way better than kafka.

- Read scaling: Bookkeeper uses consensus to store log records, which means you can read from whatever shard you want. Whereas in kafka, reader has to connect to partition master.

Re:Apache Kafka Goes 1.0 by petermelias on Wednesday November 01, 2017 @08:04PM (#15606685)
To clarify the use of the term "rebalancing" here for people who are familiar with Kafka's terminology: this is (I believe) referring to partition _storage_ rebalancing (redistribution of topics replicated across brokers). It is _not_ referring to Kafka's notion of rebalancing across a consumer group subscribed to a set of topic partitions.

The specific concern here is the possibility that Kafka's ISR strategy can potentially result in a corrupt leader partition and truncate messages to recover from a broker machine failure. The unclean leader election configuration setting for Kafka brokers is relevant here.

Also "better" is subjective depending on your configuration, requirements, and storage backends.

Re:Apache Kafka Goes 1.0 by lmsp on Wednesday November 01, 2017 @08:55PM (#15606944)
another point added to 'rebalancing' -- when kafka rebalances the partitions, it has to copy all the data for the partitions that are moved around. it might not be a big problem when retention is small. however it is pretty worse when retention period is longer, rebalancing is going to exhaust all the bandwidth (both network and I/O) in the cluster. people don't realize the fact until they want to grow the cluster (adding more brokers) to support increased traffic.