There is currently (much anticipated!) work underway to remove Kafka's Zookeeper dependency: : https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A...

As of v0.9 (about 4 years ago), clients have no use for a direct connection to Zookeeper, and don't maintain one.

Fat clients are more effort to make, which is a significant downside, but this is abstracted from the user and Kafka has critical mass - the client story outside Java is improving dramatically. A fat client allows for optimal availability, throughput, latency.

Thanks, that's promising.

A fat client makes a lot of sense, but if it's written in Java, then only JVM-based languages benefit. A client written in C, C++ or Rust — or any other language with a similarly good C-interop story — could easily be consumed by just about anything except, ironically, Go.

there are pros and cons to having a C dependency. the main benefit is reliability (get it right once, leverage that across languages). Another benefit is usually performance. The main practical gotcha is that security dependencies are incompatible across platforms, and the best you can do is distribute a number of builds of the C dependency compatible with common platforms (but that is workable).

the go story isn't as bad as you might think. there is a performance hit with the interop, which is of the order of 20% iirc - but throughput still massive, and this is not of practical significance. The Confluent go client (a binding to librdkafka), is used to push much of the volume through two of the largest Kafka installations in existence.

there is a PR open on the confluent go client that will include builds of librdkafka in the github repo, meaning this will 'just work', without the need to install librdkafka separately in most common scenarios.

note: i work on kafka clients (including minor contributions to the confluent go client).

Thanks -- I thought the Go stack switch overhead was more than 20%, that's not awful. A lot better than JNI, I think?

> I particularly like your idea that — if I understand it correctly — data can be stored independently of the log itself.

Not quite what I was getting at, though I see what you mean. (You might be interested in Badger₀, and the WiscKey paper₁ it's based on.)

[0] https://github.com/dgraph-io/badger

[1] https://www.usenix.org/system/files/conference/fast16/fast16...

To me, a "log" is just a directory of files (= log segments), where only one file (the newest segment) is ever opened for writing (specifically appending), and those appends are only ever of whole, valid records. Given a "log", you can treat everything except for the newest file/segment in it as an object (i.e. object-store backed local cache file, with atomic insert/replace/delete), rather than a file (= inode containing buffer of extents) per se.

Mind you, in Kafka's design, all the old segments get "compacted" together, meaning that they get converted into a single on-disk hash table of the newest value for each key. That's cool and all, but the compacted-log hash table doesn't have object-like semantics; it suffers constant random writes every time new segments get "compacted into" it.

IMHO, rather than this kind of compaction, I'd much rather just have streaming compaction, sort of like LevelDB's level-based compaction. Ideally, every time a broker finalized a segment that caused it to now have 2^N contiguous historical segments of level N in a log, it would queue up a streaming-compaction job into another topic; and then some arbitrary node would work that job by streaming through the relevant segments with a de-duping merge sort, producing a new combined segment of level N+1 and then purging the level-N source segments from the object store.

I had to read your earlier comment a couple of times, and I think I understand! When you said object store, I was thinking something like S3, which wouldn't be sufficient for the semantics you outline.

I don't know the internals of Kafka too well, so I'm surprised it doesn't already do what you suggest. An LSM tree system would be a more natural fit in my mind than fixed-sized segments. I wasn't thinking of compaction at all — I had to read a bit about why Kafka compaction wasn't just about deleting log entries beyond the retention horizon.

I suspect that in any Kafka-type application, one of the main challenges is balancing the storage, access performance and consistency of "old" and "new" objects. The head of the stream tends to be where contention is, everything else is read-only and "baked" with indexes and other mechanisms that make range searches quick, and can be heavily cached with long TTLs.

The reason I mentioned external data storage is that if the data never mutates, but it is the log structure which is mutable, then your log is just a sequence of metadata:

  ["123456", "s3://mybucket/xyz1"]
  ["123457", "s3://mybucket/xyz2"]
  ["123458", "s3://mybucket/xyz3"]

Then you can optimize for the structural aspect of the database — you can have very fast data store handling the sequencing of the data and the querying of ranges and so on — and simply offload the physical storage of the log entry data to something hyperoptimized to take care of large blobs efficiently.

I've not thought about what the overhead of storing many very small objects in something like S3 would be. Like if you asked for a range of log entries from one hour back and it's 1 million objects, you can certainly parallelize the fetching efficiently, but you'd also be incurring 1 million S3 requests and doing quite a lot of HTTP traffic (nothing that a local LRU cache couldn't help you with, but still).

So it's probably worth merging them — so you could have a two-level system where newly inserted objects lived in a "fresh" store and then got slowly merged into bigger chunks that would be offloaded to a "frozen" store.

I wouldn't mind sitting down and trying to build something like this in Go using Badger.

I’m confused on your JVM heap concerns. Not trying to call you out, just want to understand. You should never need to provision more than 6GB of heap for a Kafka JVM. 90% of the OS memory leftover it will use for file handlers / page cache which is all buffered and which you don’t need to manage. Zookeeper JVM you might need to worry about but I’m running 5k partitions with hundreds of consumers just fine on 6GB heap on those as well. I use on average like 400MB. It’s not like running ElasticSearch.

Resource consumption matters a lot more in a small, self-funded company that can't just throw boxes at a problem.

My data from non-JVM apps is that we can cram a whole lot of them into small 16GB VMs via Kubernetes — Go apps especially. With JVM apps, we typically have to dedicate larger 24-32GB boxes to 1-2 apps because they absolutely need the legroom. Since RAM isn't "elastic" like CPU is, you get less flexibility in managing big pieces than with small ones, especially big pieces that must stay up. We have to preallocate a certain amount and watch our limits.

I've never actually run ZK or Kafka in production, so this is pure speculation, but I didn't get any fuzzy feelings from playing around with a test cluster recently. When you combine ZooKeeper with Kafka we're probably talking about 3+ dedicated boxes because they don't fit into existing ones.

These things add up, and right now I'm trying to reduce our monthly GCP bill, not add to it!

(We do run Elasticsearch. Its memory use seems particularly egregious, indeed. I'm sure these other things are lighterweight.)

They are in the process of removing the Zookeeper dependency from Kafka: https://cwiki.apache.org/confluence/display/KAFKA/KIP-500:+R...

Have you looked at Liftbridge? https://github.com/liftbridge-io/liftbridge

Yes, it looks promising. But it's in "early" development and not production-ready, according to its author.