ZooKeeper alternative with:

1. Snapshots and logs take much less amount of space on disk due to better compression.

2. No limit on the default packet and node data size (it is 1 MB in ZooKeeper)

3. No zxid overflow issue (it forces restart for every 2 bn transactions in ZooKeeper)

4. Faster recovery after network partitions due to the use of a different distributed consensus protocol.

5. It uses less amount of memory for the same volume of data.

6. It is easier to setup, as it does not require specifying the JVM heap size or a custom gc implementation.

7. A larger coverage by Jepsen tests. (This could be hard to believe, but true - ZooKeeper is tested by Jepsen, but Keeper takes the existing tests and adds more).

8. The possibility to store snapshots and previous logs on S3.

C++ isn't a key detail, just a consequence of the fact that the main ClickHouse code base is written in C++.

If you need a distributed consensus system but not necessarily compatible with ZooKeeper, there are plenty of options: Etcd, Consul, FoundationDB...

2 is configurable (with specific caveats) 3 I believe was solved a very long time ago? 6 is this remotely relevant to java apps? You still want limits in cloud delays so not sure how this is remotely a consideration.

Disclaimer: I working on this blog at ClickHouse with the team.

We'll look in to them and adding to some of our social promotion over the coming weeks. Will try to find a way to give you credit.

Disclaimer: I worked on this blog with the team at ClickHouse.

I like your suggestions!

Some of the benefits we summarized in this summary page https://clickhouse.com/clickhouse/keeper include ease of setup and operation, no overflow issues, better compression, faster recovery, (dramatically) less memory used, etc..

There was actually a reason why C++ was important for us at ClickHouse, and it's because C++ is our main code base and managing a Java project as part of it was not natural, but you right - for standalone use of this alternative, that doesn't matter.

How much did memory safety factor in the decision? I mean only last week the IT world was yet again bitten by a major memory safety bug, in libwebp.

Given that their entire code base is written in C++, and switching to a different language would be a significant retooling for the team, I think it's reasonable to assume that it did not. Language choice is rarely made on the grounds of specific features, and is often made on the grounds of ergonomics and team knowledge.

A more revealing question is, "How are you dealing with memory safety in this implementation?" There are ways to improve memory safety in C++ through tooling and idiomatic style. Are these things being used?

Keeper is tested in the same way as ClickHouse.

There are Keeper only tests, but we run ClickHouse with Keeper for all of our server tests.

For each test we try to use all useful tools for verifying safety and correctness like sanitizers.

E.g. an interesting tool we introduced in our codebase for thread safety https://clang.llvm.org/docs/ThreadSafetyAnalysis.html#

We found some issues using sanitizers in our codebase and NuRaft library itself which were instantly fixed.

And let's not forget about Jepsen which showed some really tricky bugs but were more related to the correctness.

Thanks. That is useful.

I would suggest looking into CBMC and similar tools as well. Model checking is incredibly useful.

Sadly I never put enough effort into trying out such checks. but your excitement about them gives me motivation to properly try them out.

CBMC is subtle and will require some code changes to use effectively.

The real key for using it, in my opinion, is to isolate individual classes and functions. Avoid instrumenting code with recursion and loops, and focus on defining and verifying function contracts, class invariants, and resource / memory lifetimes.

It will require a significant amount of work to mock up standard library and third party library APIs, but the real beauty of CBMC is that once you define the interface contracts for these APIs and libraries, you can verify every use of them.

I used CBMC previously to verify proper usage rules with C / JNI integration. JNI can be one complicated beast, and CBMC handily managed rule checks for its use.

I'm an extremely careful developer who unit tests everything and strives for 99% coverage. CBMC was still able to detect a memory overwrite flaw in a networking library I wrote that was based on undefined behavior due to integer promotion and offset math. This passed the various sanitizers and unit tests I had in place, but CBMC was able to reduce it to an actual crash condition that was potentially exploitable.

I don't think I can over-emphasize the usefulness of this tool.

It is to let you know to not use it, since it is a pile of Memory related CVEs just waiting for the joy of discovery.

That's true - C++ libraries are typically bug-ridden and require exhaustive efforts to clean up.

But the latest bugs found by ClickHouse continuous integration system in the related library were fixed about a year ago:

https://github.com/eBay/NuRaft/pull/373 https://github.com/eBay/NuRaft/pull/392

If even Google, Mozilla, Apple and Microsoft can't get C++ right in their apps with all the fuzing tools they invented I don't have much faith other companies are going to fare better with memory safety and C++.

I'd be most interested in a Zookeeper alternative that doesn't have massive bugs in leader election.

Since the article states that they're using Raft and not ZAB for the consensus algorithm and leader election, it must be less prone to bugs when it comes to electing a leader. Since Raft is easier to reason about and the leader election process is more straightforward (Raft minimizes the chance that any two nodes will be candidates at the same time and thus avoids starting multiple concurrent elections).

Just adding on to a nice response from abronan, our internal protocol also does some optimizations when it comes to leader election, e.g. Pre Vote protocol (https://github.com/eBay/NuRaft/blob/master/docs/prevote_prot...)

Also, we apply many different faults in our Jepsen tests which are run 3 times a day and we never had a problem with leader election. I know this doesn't confirm that there is no bug in it but it's pretty reassuring I would say.

Java vs C++ is very important implementation detail, especially for 'the benefits to the user'. Java is commercial platform which requires a fee to Oracle if used in enterprise, while C++ complied binary does not.

That's a _very_ incorrect statement. You can use any OpenJDK (which is GPLv2 with classpath exception) distribution you want to run Apache Zookeeper without having to have any agreement with Oracle or pay any fee. The Oracle JDK is just Oracle's commercial version of their OpenJDK distribution with Oracle support.

You can use the OpenJDK distro shipped in your Linux distro (RedHat, Debian, etc.), you can use Microsoft's OpenJDK distro[1], you can use the Eclipse OpenJDK distro, you can use Amazon's OpenJDK distro [3] and there are a whole bunch more.

[1] https://www.microsoft.com/openjdk [2] https://adoptium.net/ [3] https://aws.amazon.com/corretto/

The usual FUD, no it doesn't require any fee, use OpenJDK.

Several distributions to chose from.

Or you can just use OpenJDK, right?

All ZooKeeper installations I've seen so far in production were on Oracle Java JDK for one or another reason.

Probably because people are stuck in the 2010's. OpenJDK used to have more compatibility issues.

What year are you from to say such nonsense?

Running services on the JVM is terrible and requires much more resources than other platforms

Depends on how much one cares about memory corruption, developer tooling and library ecosystem.

Yes I agree memory safety is an important trade off, but the performance wins can be worth it in some cases.