Re:Kafka is Fast – I'll use Postgres
by LgWoodenBadger on Thursday October 30, 2025 @01:38PM
(#45762748)
I don't think "while retaining ordering" is something you want to include here, since you can't guarantee processing order for any MQ system without serializing the consumption down to a single consumer.
Re:Kafka is Fast – I'll use Postgres
by rileymichael on Thursday October 30, 2025 @02:58PM
(#45763875)
absolute order, you're correct. but several systems support demultiplexing an interleaved stream by some key so that order is retained for all messages in each key space (pulsar key_shared subscriptions, azure service bus sessions, etc). you're still serializing consumption / have an exclusive consumer for each key, but with global parallelism.
here's the equivalent in parallel-consumer: https://github.com/confluentinc/parallel-consumer?tab=readme...
Re:Kafka is Fast – I'll use Postgres
by mwarkentin on Thursday October 30, 2025 @09:02PM
(#45767269)
There is a whole KIP that is “preview” on Kafka 4.1 to handle this use case natively: https://cwiki.apache.org/confluence/plugins/servlet/mobile?c...
Note: I haven’t had a chance to test it out in anger personally yet.
Re:Kafka is Fast – I'll use Postgres
by rileymichael on Thursday October 30, 2025 @09:39PM
(#45767456)
yep i mentioned that in another thread below. very excited to see it added, but it will be some time until its production ready / they implement the full feature set
you really can't. getting per-message acks, dynamically scaling competing consumers without having to repartition while retaining ordering, etc. requires a ton of hacks like client side tracking / building your own storage on top of offset metadata / etc.. and you still won't have all of the features actual message queues provide.
to make it worse, there is very little public work/discussion on this so you'll be on your own to figure out all of the quirks. the only notable example is https://github.com/confluentinc/parallel-consumer which is effectively abandoned