News for hackers. Stuff that matters.
Re:Kafka without ZooKeeper by nemothekid on Tuesday March 30, 2021 @08:30PM (#26642202)
Before Raft (2013), if you wanted reliable, consistent distributed metadata store you had to implement Paxos which is notoriously difficult to get right. Every service that needed some type of leader election or highly consistent store let Zookeeper deal with that problem (Mesos, Spark, Druid, Storm, and a ton others).

After Raft, it became easier to just implement that layer yourself and so most projects after Raft (or probably more accurately once people started seeing how stable etcd was, ~2014), just used Raft internally where they would have previously used zookeeper.

Re:Kafka without ZooKeeper by dikei on Tuesday March 30, 2021 @09:31PM (#26642565)
To be fair, many project's Raft implementations contained errors that can and had lead to data lost, so it's not all sunshine and roses.

IMHO, it's still easier to delegate the consensus problem to a third party service like Zookeeper or ETCD.

Re:Kafka without ZooKeeper by nvarsj on Wednesday March 31, 2021 @09:35AM (#26647069)
Raft is a large improvement over Paxos for practical implementations. But it's still tricky to get right. As far as I know, the only widely used, battle tested Raft implementation is github.com/hashicorp/raft. Which is why so many distributed systems are being built on golang over the last few years. I don't know if there is any Java raft implementation which has reached that level of maturity yet - but it seems like Confluence is trying with kraft.
Re:Kafka without ZooKeeper by mavelikara on Wednesday March 31, 2021 @01:56AM (#26643976)
What are the notable projects that implement Raft internally for leader election?

Also, do any of those projects publish their Raft implementation as a library for other projects to include?

Re:Kafka without ZooKeeper by beberlei on Wednesday March 31, 2021 @02:13AM (#26644036)
Anything from hashicorp, vault, consul, nomad for example. yes there is a go library for the basic raft setup afaik