lyha

Kafka stores everything to disk, this may not be what you are looking for for your RPC calls (that you would make usually as a direct service to service HTTP call). Moreover kafka "topics" are statically declared (i.e. by admin scripts instead of a public API), and it's a heavy-weight operation. So it's not the best fit to have micro-services registering themselves and dynamically creating "topics" for each method.

Actually Redis returns the number of consumers that were subscribed during your publish and received the message, so you can detect that no service is available and return your 404 status. The real limitation for RPC over Redis pub/sub to me is that you can only broadcast your messages to all consumers, and not have them load-balanced between the subscribed applications. So you end-up following a discover/unicast call pattern where Redis doesn't bring much to the table for the actual RPC call. Maybe that's an opportunity for a cool addition to redis...

MQTT is a protocol with public specification for lightweight client / message broker communications, allowing publish/subscribe exchanges. Multiple implementations of client libraries and brokers (Mosquitto, JoramMQ...) exists and are virtually compatible. MQTT just specifies the transport, and vaguely the application part (i.e. how data is handled and possibly stored, how clients are authorized...). The spec is not clear if data consumed on a topic is only real-time or possibly persisted. The spec doesn't state anything about how the message broker implementing MQTT could/should scale.

On the other hand, Apache Kafka is a message broker based on an internal "commit log": its focus is storing massive ammount of data on disk, and allow consumption in real-time or later (as long as data is still available on disk). It's designed to be deployable as cluster of multiple node, with good scalabily properties. Kafka uses its own network protocol.

So you are comparing two different things here: a standard pub/sub protocol (with multiple implementations), and a specific message storing/distributing software, vaguley of the same family with its own protocol.

I'd say that if you need to store massive amount of messages, to ensure batch processing, look more at Kafka. If you have lots of clients/apps exchanging messages in real-time on many indpendants topics look more at the MQTT (or even AMQP) message brokers implementations.