News for hackers. Stuff that matters.
Re:Apache Kafka Goes 1.0 by manigandham on Wednesday November 01, 2017 @01:11PM (#15603353)
Congrats to the kafka/confluent team.

Side note: https://pulsar.apache.org also seems to be gaining traction and has a much better story around performance, pub/sub, multi-tenancy, and cross-dc replication. Will be interesting to see the evolution of both going forward.

Re:Apache Kafka Goes 1.0 by delaSoul on Tuesday November 14, 2017 @12:14PM (#15696980)
Has somebody compared pulsar with pravega(http://pravega.io)?
Re:Apache Kafka Goes 1.0 by lmsp on Wednesday November 01, 2017 @04:40PM (#15605275)
another side note, pulsar supports kafka api since 1.20.0 - https://github.com/apache/incubator-pulsar/releases/tag/v1.2...
Re:Apache Kafka Goes 1.0 by slap_shot on Wednesday November 01, 2017 @03:11PM (#15604542)
Interesting. Can someone give a quick/high level list what makes it better than Kafka and in what use cases?
Re:Apache Kafka Goes 1.0 by manigandham on Wednesday November 01, 2017 @03:50PM (#15604853)
I recommend reading the Pulsar architecture page: https://pulsar.apache.org/docs/latest/getting-started/Concep...

Biggest difference is that Pulsar splits compute from storage. Pulsar brokers are stateless services that manage topics and handle clients (publishers and consumers) while reading/writing data from a lower tier of bookies, which is the Apache BookKeeper project for low-latency real-time storage workloads.

Better performance and easier scaling compared to Kafka's partition rebalancing. It also natively supports data center replication and multiple clusters working together globally. Pulsar addressing is property/cluster/namespace/topic so multi-tenant isolation is built-in.

Anywhere you use Kafka can use Pulsar and also consolidate pub/sub only systems that need lower latency than Kafka.

Re:Apache Kafka Goes 1.0 by lmsp on Wednesday November 01, 2017 @09:07PM (#15607012)
I think it is beyond partition rebalancing. There is a fact that people didn't realize of making message broker `stateless`. It is actually much better on reacting to failures or shifting traffic, which is critical when running a messaging bus for online services. because it doesn't have to wait for copying the data of a whole partition when error occurs.
Re:Apache Kafka Goes 1.0 by manigandham on Wednesday November 01, 2017 @11:04PM (#15607515)
Agreed, the architecture page and other blog posts do a more thorough job of explaining the details. Having a stateless broker layer on top of a focused data layer makes all the operations much easier. I expect BookKeeper to further integrate with the various cloud storage APIs so it can also start to become stateless cache.
Re:Apache Kafka Goes 1.0 by majidazimi on Wednesday November 01, 2017 @04:34PM (#15605217)
The biggest differences:

- rebalancing: Bookkeeper (which is the storage backend of Pulsar) manages rebalancing way better than kafka.

- Read scaling: Bookkeeper uses consensus to store log records, which means you can read from whatever shard you want. Whereas in kafka, reader has to connect to partition master.

Re:Apache Kafka Goes 1.0 by petermelias on Wednesday November 01, 2017 @08:04PM (#15606685)
To clarify the use of the term "rebalancing" here for people who are familiar with Kafka's terminology: this is (I believe) referring to partition _storage_ rebalancing (redistribution of topics replicated across brokers). It is _not_ referring to Kafka's notion of rebalancing across a consumer group subscribed to a set of topic partitions.

The specific concern here is the possibility that Kafka's ISR strategy can potentially result in a corrupt leader partition and truncate messages to recover from a broker machine failure. The unclean leader election configuration setting for Kafka brokers is relevant here.

Also "better" is subjective depending on your configuration, requirements, and storage backends.

Re:Apache Kafka Goes 1.0 by lmsp on Wednesday November 01, 2017 @08:55PM (#15606944)
another point added to 'rebalancing' -- when kafka rebalances the partitions, it has to copy all the data for the partitions that are moved around. it might not be a big problem when retention is small. however it is pretty worse when retention period is longer, rebalancing is going to exhaust all the bandwidth (both network and I/O) in the cluster. people don't realize the fact until they want to grow the cluster (adding more brokers) to support increased traffic.
Re:Apache Kafka Goes 1.0 by colordrops on Wednesday November 01, 2017 @09:32PM (#15607133)
Would you say that pulsar is strictly better than kafka, or that there are cases where kafka is preferred?
Re:Apache Kafka Goes 1.0 by manigandham on Wednesday November 01, 2017 @10:57PM (#15607486)
Yes, right now Pulsar does everything Kafka does but easier, faster, and with more features. It's especially great in combining message queuing with low-latency pub/sub.

It's also been around for years but recently open-sourced so community is smaller than Kafka, however it's growing quickly along with the usual ecosystem of drivers, extensions, and services. Unless you're already running Kafka, I would look at Pulsar first for new projects.

A great overview of Pulsar history with plenty of links is here: https://streaml.io/blog/messaging-storage-or-both/

Re:Apache Kafka Goes 1.0 by tkanos on Thursday November 02, 2017 @05:26AM (#15608920)
Unfortunately pulsar does not have a system such kafka stream. In pulsar we need to use spark to do the same (https://pulsar.incubator.apache.org/docs/latest/adaptors/Pul...).

And pulsar does not provide exactly once neither, that is something very important.

Re:Apache Kafka Goes 1.0 by lmsp on Thursday November 02, 2017 @12:27PM (#15611856)
Kafka streams is great. But it is a processing library. It isn’t an apple-to-apple comparison to a messaging system like Pulsar. First there are already many mature stream processing engines and libraries (spark, flink, heron, storm). They have been in production for years and it made no sense to write a new one without a good reason. Second, kafka stream could have done a better job, not just tight with kafka. Technically kafka streams isn’t really a very specific implementation or design to kafka. If confluent has done a better job on abstraction, I would expect it is very easy to plugin different messaging systems or log storages to run "kafka streams". Although I am not sure Confluent want to see that happen.

“Kafka has exactly-once delivery but messaging system x/y/z doesn’t provide it” is also confusing and misleading. Exactly-once is technically effectively-once: “at-least-once” and make the processing of the messages idempotent or “de-duplicated”. This has already been done in the industry for many decades in many mature stream processing engines like heron, flink. It isn’t a really new thing. And many messaging systems like pulsar already provides those primitives (e.g. idempotent producing, at-least-once delivery) for processing jobs to achieve effectively-once very easily. Streamlio folks did a great job about explaining exactly-once and effectively-once. It is worth checking this blog post out -- https://streaml.io/blog/exactly-once/

I think Pulsar itself as a distributed messaging system does provides all the three delivery semantics: at-most-once, at-least-once and effectively-once. It is very easy for people to use and integrate. I don’t think it is difficult to make kafka streams run with pulsar technically. The question is more is there a value to do that, do kafka folks wanna to do that, can the collaboration happen in the ASF?

that's just my two cents.

Re:Apache Kafka Goes 1.0 by manigandham on Thursday November 02, 2017 @10:26AM (#15610655)
Kafka Streams is just a small wrapper library to make it easy to work with Kafka partitions, along with some utilities for reading, transforming, and writing to another stream in a single step.

Pulsar doesn't have any client overhead since it's all tracked on the broker so there's no real need for a separate library. Read, process, and push messages using any code you want, 1 at a time or in batches.

There's no realistic "exactly once" either, it's idempotency or some local cache of processed events used to dedup, you can read more here: https://streaml.io/blog/exactly-once/

Re:Apache Kafka Goes 1.0 by d__k on Thursday November 02, 2017 @08:31AM (#15609765)
Is it possible to use Kafka Streams wtihout Kafka by listening and processing events from another source?
Re:Apache Kafka Goes 1.0 by tkanos on Thursday November 02, 2017 @10:07AM (#15610472)
I would say no, because kafka stream use the internal kafka method of partitions to be able to scale.
Re:Apache Kafka Goes 1.0 by lmsp on Thursday November 02, 2017 @12:35PM (#15611932)
I do think it is technically possible base on my understanding of pulsar

- pulsar provides an failover subscription mode, which seems to be the equivalent of partition rebalancing of consumer group in kafka. https://pulsar.incubator.apache.org/docs/latest/getting-star...

- it has partitioned topics as well.

- it supports idempotent producing and have effectively-once delivery semantic.

It seems to have all the kind of primitives for kafka streams. to use.