News for hackers. Stuff that matters.
Re:Alchemy Micro-Service Framework: Using RabbitMQ Instead o... by StreamBright on Tuesday March 15, 2016 @03:40AM (#11288081)
Would Kafka be a better option in this case? There might be some properties I am not aware of that makes it impossible.
Re:Alchemy Micro-Service Framework: Using RabbitMQ Instead o... by atombender on Tuesday March 15, 2016 @11:21AM (#11290284)
No, Kafka is completely unsuitable for RPC, for several reasons.

First, its data model shards queues into partitions, each of which can be consumed by just a single consumer. Assume we have partititions 1 and 2. P1 is empty, P2 has a ton of messages. You will now have one consumer C1 which is idle, while C2 is doing work. C1 can't take any of C2's work because it can only process its own partition. In other words: A single slow consumer can block a significant portion of the queue. Kafka is designed for fast (or at least evenly performant) consumers.

Kafka's queues are also persisted on disk, which is terrible for RPC.

Think of Kafka as a linear database that you can append to and read from sequentially. Its main use case is for data that can fan out into multiple parallel processing steps. For example, a log processing system that extracts metrics: You feed the Kafka queue into something like Apache Storm, which churns the data and emits counts into an RDBMS (for example).

Re:Alchemy Micro-Service Framework: Using RabbitMQ Instead o... by lyha on Tuesday March 15, 2016 @09:02AM (#11289250)
Kafka stores everything to disk, this may not be what you are looking for for your RPC calls (that you would make usually as a direct service to service HTTP call). Moreover kafka "topics" are statically declared (i.e. by admin scripts instead of a public API), and it's a heavy-weight operation. So it's not the best fit to have micro-services registering themselves and dynamically creating "topics" for each method.