1. That's untrue. Partitions are limited to what a machine may handle, but topics may be scaled across many ordered partitions.

2. This is generally handled by the client library transparently. Have you ever needed to manage this state manually?

Re 2, have you ever tried using Kafka with a non-JVM client?

Yep, I have, no issues with the consumer/producer knowing who the partition leader is.

That said, curious to hear your experiences :)

I don’t have issues per se as long as I stick to librdkafka but even that is constantly playing catch up.

Outside of librdkafka and jvm client, it’s gloves off.

Yeah, that's true, using librdkafka from C# I hit a few issues where librdkafka was somewhat behind Java in terms of features, I think the one I hit was multi-topic subscriptions.

IIRC Confluent has started putting resources into it - I would hope so, given how .NET Core is going.

That said, the state of Pulsar clients outside of the official Java ones was far worse, I was looking into .NET Core ones and the "official" one (Pulsar-DotPulsar) lacked some key features, whereas a third party one, pulsar-client-dotnet, had far more features, but was still somewhat behind the Java clients.

Caveat is that I looked into all of this when Pulsar was at version 2.6, it's not at 2.7.1, so my comments may well be out of date.

kafkacat is excellent.

https://github.com/edenhill/kafkacat

Definitely, if all one needs is command line.

for #1 if you need message order to be maintained you cannot use more than 1 partition. for #2 99% of the production issue we had where caused by bugs in librdkafka not talking to the right brokers. Also it mean that the IP for the broker need to all be public IP and the writing throughput and latency is limited by the capacity of the machine that is the master for the partition you are trying to write. If the machine hosting you partition become overloaded you have to switch the master for that partition to another machine unlike pulsar this is not done automatically and also if you replication factor is 3 your choice are limited to 2 other machine if you don't want to copy the whole partition to a new machine which would take hours.