You can also wrap the smart clients and the dumb messaging in a "dumb client" api if you so wish
I pushed against knative in our company particularly for that reason. Like we wanna use kafka because [Insert kafka sales pitch], but we don't want our developers to utilize any of the kafka features. We're just gonna define the kafka client in some yaml format and have our clients handle an http request per message. It didn't make sense to me.
Its just an abstraction layer to make things less effort.
It's one thing to have an abstraction for kafka in your code, it's another to wrap the client in a smart client that reimplements something like rabbitmq, and much worse a smart service.
Im not saying to expose the same primitives - what would be the point in that? I am saying that EVERY lib you use will be using the standard lib or some abstraction of it to perform its own utility.
> It's one thing to have an abstraction for kafka in your code, it's another to wrap the client in a smart client, and much worse a smart service.
That abstraction is exactly what i am talking about. Why write 50 lines of boilerplate multiple times throughout your code when you can wrap that up in a single function call and expose THAT as your client. You know thats exactly what you will end up doing on any non-trivial project. Or you could use a lib that already does that - such as the "official" kafka streams lib.
This is decent advice, IMO. The cost of dependency management is often vastly understated.
The cost of dependency management may be understated but it’s always less than the cost of reimplementing everything found in established libraries.
Also, let's be honest: The phalanx of developers that violently argue for importing everything and never implementing anything yourselves is way bigger than people who argue for the opposite; I don't think we need to worry about the latter making things worse as much as the former.
We've seen what the world turns into in both scenarios, I would argue, and at least with the first one we got software that ran decently and developers who knew how to actually do things. We have overall much safer languages now so their home grown solutions won't have the same safety issues that they've historically had.
With the importer crowd we've gotten software that feels like molasses and an industry full of people who know only the surface of everything and more importantly become tied to frameworks and specific APIs because they never actually go beyond any surface APIs.
As with most tradeoffs in software there is a good middle-ground, but we won't ever get there if we don't have people who argue for making things yourselves.
I can imagine home grown libraries having inconsistent api with wild and wonderful assumptions and beware the edge cases
How do you do, for example, a queue with priorities client side without it being insanity? That's a relatively basic AMQP thing. Or managing the number of redeliveries for a message that's being repeatedly rejected.
You can absolutely try to build some of this with a look-aside shared data store that all clients have to depend on in order to emulate having the capability in the broker, but you just introduced another common point of failure in addition to the messaging infrastructure. Life is too short for this.
I'm just saying you can "dumb down" the client side on kafka by creating an abstraction layer (or one of the many higher level libs that already do that).
- RabbitMQ: Smart messaging, dumb client - Kafka: Dumb messaging, smart client
Have you heard of Fluvio? Fluvio: Smart messaging, Smart Client, Stateful Streaming
Kafka + Flink in Rust + WASM Git Repo - https://github.com/infinyon/fluvio
The problem is you cannot mark individual messages as read, for a given consumer&partition you can only update the offset for a partition.
If a certain message processing takes very long, all other messages in that partition will have to wait.
Also, with kafka, the max read concurrency is equal to the number of partitions, for something like rabbitMq it is much higher; but you do get nice message ordering for any given partition in kafka which you do not get in RabbitMq (afik); you are also get some really nice data locality with kafka because unless the consumers get the partitions re-assigned, all messages for the same key are served on the same physical consumer.
Hence "smart clients". If you MUST process every message at least once, you will anyway be tracking messages individually on the client (e.g. a DB or file system plus logic for idempotent message processing) and thus disable auto-offset commits back to the cluster for your consumer.
RabbitMQ says "let me track this for you", Kafka says "you already need to track this so why duplicate the data in the cluster and complicate the protocol".
If you don't have reliable persistent storage available and insist on using the Kafka cluster to track offsets, you can track processed offsets in memory and whenever your lowest processed offset moves forward, you have your consumer commit that offset manually as part of its message loop.
If your service restarts your downstream commands need to be idempotent of course because you will reconsume messages you may have previously processed, but this would be the case regardless of Kafka or RabbitMQ unless you're using distributed transactions (yuck).
> If a certain message processing takes very long, all other messages in that partition will have to wait.
You can stream messages into a buffer and process them in parallel, and commit the low watermark offset whenever it changes, as described above. I've implemented this in .NET with Channels and saturate the CPUs with no problem.
And there are libraries that will manage all this for you e.g. https://github.com/line/decaton
I've seen databases used as messaging queues and if it was up to me, I'd never do that. It's usually "but we already have kafka + db, why burden ourselves with another messaging technology?", which is fair.
> You can stream messages into a buffer and process them in parallel, and commit the low watermark offset whenever it changes, as described above. I've implemented this in .NET with Channels and saturate the CPUs with no problem.
That is very nice -- certainly seems better than just batch processing of kafka messages, but you're still just kicking the can down the road. How large do allow the buffer to become and what do you do when it's getting too large?
You probably use a DLQ.
Don't get me wrong, I think the buffer idea probably works most of the time.
Regarding the buffer size: you need to implement back pressure, especially if you are CPU and not IO bound; it's another thing that's easy to get wrong with Kafka.
It combines the best of both Kafka and RabbitMQ IMO.
Looking at Wikipedia (https://en.wikipedia.org/wiki/NATS_Messaging) I see that I'm technically right, it's JetStream that does the storage layer - but it's part of the NATS server.
From memory, I really liked the philosophy of NATS but found the nomenclature confusing.
I have used it mostly for message-first services, and found subject-based messaging a breath of fresh air to decouple services. You can do the same thing with RabbitMQ topic exchanges, but it requires quite a bit more hand-waving.
So you can for example ask for [the first/the last/all] of the messages on a particular subject, or on a hierarchy of subjects by using wildcards. All the filtering is done at the server level.
You don't want to classify them by what they do. You want to classify them by what the clients must do/experience.
I think this is an odd way of putting it. One is smart messaging; dumb clients. The other is dumb messaging; smart clients. It turns out the latter (i.e. Kafka) scales wonderfully so you can send more data, but you add complexity to your clients, who can't just now pluck messages off a queue to process, or have messages retry upon the first 3 failures, as they could with RabbitMQ.
Having said that, Kafka lets you keep all your data, so you don't have to worry about losing messages to unexpected interactions between RabbitMQ rules. But having said that, now you have to store all your data.