News for hackers. Stuff that matters.
Re:Apache Pulsar is an open-source distributed pub-sub messa... by eerrt on Thursday January 02, 2020 @10:58AM (#21936415)
How does this compare to Redis Pub-Sub or RabbitMQ?
Re:Apache Pulsar is an open-source distributed pub-sub messa... by sz4kerto on Thursday January 02, 2020 @11:04AM (#21936483)
Very different. Pulsar is primarily a Kafka competitor.

- it is much more performant than RabbitMQ - it's a commit log as well, not just a pub-sub system, ie. it is a good candidate as the storage backend for event sourcing - it supports geodistributed and tiered storage (eg. some data on NVMe drives, some on a coldline storage) - it's persistent, not in-memory (primarily)

.. and so on.

Re:Apache Pulsar is an open-source distributed pub-sub messa... by EGreg on Thursday January 02, 2020 @12:00PM (#21937078)
What about ZeroMQ?

Why use RabbitMQ and Kafka if you can use ZeroMQ? Meaning, isn’t it far more performant and distributed?

Maybe I am missing something here.

Re:Apache Pulsar is an open-source distributed pub-sub messa... by Joeri on Thursday January 02, 2020 @12:11PM (#21937254)
Message queues and message logs do different things. The idea of the log is that subscribers can show up after the log is written and read or reread it from the beginning. In an event sourced architecture you use the log as the source of truth and all consumers can replay the log against a local store to reconstruct a view of the system’s state. You also can use a log for pubsub, but if that’s all you need one of the MQ solutions is usually a better fit.
Re:Apache Pulsar is an open-source distributed pub-sub messa... by sz4kerto on Thursday January 02, 2020 @12:39PM (#21937590)
> Why use RabbitMQ and Kafka if you can use ZeroMQ?

They are totally different, you're comparing apples with oranges.

ZeroMQ gives you basic, very fast tooling to communicate between distributed processes. ZeroMQ does not provide tooling for e.g. maintaining a strictly ordered, multi-terabyte event log. And so on.

Re:Apache Pulsar is an open-source distributed pub-sub messa... by EGreg on Thursday January 02, 2020 @12:44PM (#21937664)
Yes but isn’t this a bit like comparing git / bitkeeper vs subversion / perforce?

Basically, one is decentralized and you can set up a massively parallel architecture, with eg each topic or subthread having its own pubsub.

The other is a monolithic centralized pubsub architecture.

You could argue that git in large institutional projects converges to a monolithic repo so at that point it’s less efficient even than svn.

But for most use cases, ZeroMQ would allow far more flexible distributed systems topologies and solutions. No?

Edit: HN and Google are both awesome: https://news.ycombinator.com/item?id=9634925

Re:Apache Pulsar is an open-source distributed pub-sub messa... by ryeguy on Thursday January 02, 2020 @10:51PM (#21942962)
Zeromq is just a bit of sugar on top of tcp sockets. It isn't a message queue or anything close. You would be wasting a ton of time reimplementing a lot of basic features like retries, persistence, service discovery, dead letter queues, priority, and a ton of other stuff.
Re:Apache Pulsar is an open-source distributed pub-sub messa... by frogger23123 on Thursday January 02, 2020 @02:00PM (#21938507)
> You could argue that git in large institutional projects converges to a monolithic repo so at that point it’s less efficient even than svn.

Not true. Facebook and Google do not use Git. Microsoft does not use vanilla Git for their monorepo. They created this extension to make it scalable https://en.wikipedia.org/wiki/Virtual_File_System_for_Git

Re:Apache Pulsar is an open-source distributed pub-sub messa... by TuringNYC on Thursday January 02, 2020 @12:08PM (#21937192)
I went to https://pulsar.apache.org but didnt find a "Why Pulsar and not Kafka" -- is there an answer to that, or is this another Kafka competitor with the same strengths and not a specific differentiator?
Re:Apache Pulsar is an open-source distributed pub-sub messa... by cbartholomew on Thursday January 02, 2020 @12:15PM (#21937307)
Here is a two-part blog post I wrote on why Pulsar and not Kafka: https://kafkaesque.io/5-more-reasons-to-choose-apache-pulsar...
Re:Apache Pulsar is an open-source distributed pub-sub messa... by richdougherty on Thursday January 02, 2020 @01:44PM (#21938325)
Thanks! And Part 1 seems like a good place to start: https://kafkaesque.io/7-reasons-we-choose-apache-pulsar-over...
Re:Apache Pulsar is an open-source distributed pub-sub messa... by dickjocke on Thursday January 02, 2020 @12:40PM (#21937603)
Why is one Apache providing competing with another one?
Re:Apache Pulsar is an open-source distributed pub-sub messa... by spelunker on Thursday January 02, 2020 @01:15PM (#21938010)
Plenty competing Apache projects exist, Pulsar and Kafka aren't unique in that regard.

I don't think Apache cares if it's maintaining similar projects.

Re:Apache Pulsar is an open-source distributed pub-sub messa... by tyingq on Thursday January 02, 2020 @11:10AM (#21936553)
It's a persistent store, so that would be different from Redis Pub-Sub. Compared to RabbitMQ, Pulsar seems to favor strong ordering and protection from message loss.

This blog post offers some more info and leaders to other posts comparing Pulsar to RabbitMQ and Kafka: https://jack-vanlightly.com/blog/2018/10/2/understanding-how...

Re:Apache Pulsar is an open-source distributed pub-sub messa... by Joeri on Thursday January 02, 2020 @12:06PM (#21937156)
It’s closer to redis streams, except like kafka you can scale topics beyond the limits of a single server because they can be distributed. You couldn’t run the twitter firehose over redis streams, but you can run it over pulsar or kafka, given enough hardware.
Re:Apache Pulsar is an open-source distributed pub-sub messa... by qaq on Thursday January 02, 2020 @11:06AM (#21936503)
This scales to multi-datacenter deployments well. Has strong multi-tenancy support if memory serves Yahoo is running a single cluster for all of their properties.
Re:Apache Pulsar is an open-source distributed pub-sub messa... by xrd on Thursday January 02, 2020 @10:58AM (#21936421)
Or Firebase?
Re:Apache Pulsar is an open-source distributed pub-sub messa... by Eikon on Thursday January 02, 2020 @11:03AM (#21936471)
Or Kafka ?
Re:Apache Pulsar is an open-source distributed pub-sub messa... by sz4kerto on Thursday January 02, 2020 @11:05AM (#21936489)
Kafka is primarily designed for streaming, pulsar is both for streaming and queueing.

Firebase is a completely different animal.

Re:Apache Pulsar is an open-source distributed pub-sub messa... by manigandham on Thursday January 02, 2020 @04:46PM (#21940244)
Read the other comment thread: https://news.ycombinator.com/item?id=21936523
Re:Apache Pulsar is an open-source distributed pub-sub messa... by qaq on Thursday January 02, 2020 @11:07AM (#21936522)
Scales much better the storage layer is separate from brokers so you can scale things independently.