News for hackers. Stuff that matters.
Re:Apache Pulsar is an open-source distributed pub-sub messa... by bubbleRefuge on Thursday January 02, 2020 @01:02PM (#21937876)
Isn't using Kubernetes kind of an anti-pattern due to failover and rebalancing logic clashing? If Kubernetes is killing and re-starting nodes and the cluster's brokers are detecting dead brokers and rebalancing partitions as a result, it seems counterproductive.
Re:Apache Pulsar is an open-source distributed pub-sub messa... by addisonj on Thursday January 02, 2020 @01:58PM (#21938471)
This is one of the main benefits of Pulsar is that because state is split between brokers and bookkeeper and bookkeeper doesn't need re-balanced (due to it's segment based architecture where you choose new bookies with each new segment), we really don't have to worry about re-balancing (in general, not just in case of failover) of storage. It is true that topics map to a single broker, but generally, Pulsar has really good limits on memory so we don't see nodes getting killed by limits and we only really see re-scheduling for real issues.

While there certainly is some aspects you need to be aware of, generally, Pulsar is much more "cloud native" and maps quiet well to k8s primitives.

Re:Apache Pulsar is an open-source distributed pub-sub messa... by jupp0r on Friday January 03, 2020 @12:41PM (#21948256)
It's a good pattern because it regularly forces you to deal with pods/nodes going away so that the system is designed to handle this well without human intervention. There is no system where nodes don't go away because of hardware errors/updates/decommissions, so you might as well establish it as unexciting routine from the start.
Re:Apache Pulsar is an open-source distributed pub-sub messa... by rhizome on Thursday January 02, 2020 @01:06PM (#21937918)
To the degree that that conflict exists in their implementation I would think that it's possible to account for all of that.