Maybe it's just you, maybe not. It would be beneficial for you to provide some substance to your claim. Then we would have something to discuss.

Unfortunately we do not have something to discuss. All these consumer/producer/broker failing randomly, system behaving erratically is me doing it wrong. Those white papers from Confluent and testimonial from customers are proof that Kafka works fine.

It's quite beautiful, but put so simply it does look nightmarish.

"Then, the otters would decide which part of the river to put the message in"

I'd imagine a child looking at this going like "why are they doing this to the river". Why are they are throwing things into the river. :D

No, put so simply it's cute. In production it's nightmarish.

Individual Kafka consumers and producers tend to have simple behavior, which is good, and initially plugging them together yields simple, predictable systems, but people tend to keep going until they experience pain, at which point they have a system that is right at the limit of what they can understand. Then further feature work pushes them over the limit into darkness.

I think Kafka needs the equivalent of OpenAPI and Redoc, a simple spec and document generator, but for groups of consumers and producers rather than single applications. This would increase the tractability of complex systems, but it would also let you see the system getting more complex over time, even when you haven't reached the pain point yet.

https://www.asyncapi.com/ is in its infancy but is attempting what you describe.

(I view OpenAPI as a near-failure, but, good luck to everyone trying.)

Thanks for the tip; that's a really cool project. It looks like the documentation generator for it works on a single file at a time, and a single file defines a single service. I was thinking of something that works on the level of a system of interacting services. I want a tool that reads the API specifications for a group of services and documents the interactions between them. I'd like to look at service X and see that it emits message Y on queue Z, and then see which services read message Y on queue Z, jump to their documentation, etc. I think the AsyncAPI format is perfect to build on, though. I'll start there if I decide to take a hack at it.

I am not sure if I completely understand your use-case but it seems that the config can be distributed across multiple files.

https://www.asyncapi.com/docs/specifications/v2.0.0#referenc...

I'm taking a closer look today, and it looks to me like one AsyncAPI document, which can be defined using multiple files, defines one application. If you look at the "fixed fields" section under the root object, the second field is "id"[0]:

    id Identifier Identifier of the application the AsyncAPI document is defining.

This format is fine, but the tool I'm looking for is something that will read the definitions of multiple AsyncAPI documents (multiple applications) and show how their inputs and outputs connect, so I can answer a question like, "When application X publishes message Y on channel Z, which applications consume that message?"

AsyncAPI gets me 90% there by defining the service spec and providing code to parse it. It's possible somebody has already written the rest; I'll have to see.

[0] https://www.asyncapi.com/docs/specifications/v2.0.0#fixed-fi...

KSQL seems to address this gap.

Is this comment intentionally Kafkaesque?

What do you find nightmarish about it?

Kafka is very configurable .. and teams struggle to optimize the production deployment configuration and tend to second guess their decisions. Getting a managed deployment from Confluent etc. mitigates these issues as you can count on proven SMEs to deliver a well optimized solution. However, if you were to DIY , you might not be so blessed. Disclosure: I never did a DIY Kafka prod deploy -- my employer was already a Confluent customer.