toomanybits

Now Kafka has ZK taken out and tiered storage coming in these factors are becoming mute points. With tiered storage Kafka has 2 layers and hence the benefits of broker/storage decoupling. By comparison, Pulsar has broker, bookkeeper, S3, and ZK layers to contend with. This is why a second layer was never added to Kafka directly.

I think that's one of the main points. Now you can run it as a single process more like a traditional broker (although it's obviously still a log).

I've heard of Comcast, but that's Yahoo. Not heard of the others.

Being used by at least one company of significant size that (a) i've heard of and (b) isn't directly connected to the project would be a good start.

You don't need a CS degree or Martin Kleppmann's book to work out it's a GPITA.

I'm lost too. Kafka auto creates topics by default. Maybe you're referring to being able to create more topics? But that seems to be unproven. Kafka's limit is metadata and Pulsar is more metadata dependent than Kafka.

> Pulsar will happily do with just 2x. This is just wrong. Pulsar provides weaker guarantees than Kafka. It's a quorum based system. If you run with two replicas Pulsar can't provide F-1 guarantees which Kafka can.

Nope. It sucks. RIP.

Show me one

1. is true, but if you want that data to move to a new node, it still needs to be replicated. Kafka's approach is to use tiered storage (which I believe is close to completion).

2. Kafka can read from a replica node. It's relatively new but it's there.

This smacks of being heavily one-product-focussed to me. Being a Kafka user it's hard enough managing and understanding one system, nevermind three or four joined together.

Maybe it's a bit faster or a bit more elastic, or whatever, who knows. What I really care about is whether I get called at 3am and in that regard the argument seems pretty weak. Kafka for all its woes is a solid system you know you can count on.

I'd much rather see someone come up with a truly innovative alternative that actually pushes the boundaries, rather than just copying what's there already, and adding a few window dressings.

Are you sure? he seems to be using the past tense: "written so that software products like Landoop were free to embed our open source"

there's a live stream of this talk happening now https://www.youtube.com/watch?v=m-0cSdxiLLY#t=222