Is it true that a message from a queue will disappear after it is consumed successfully? If yes, at this moment, how do you make kafka topics work as queues?

Think about this for a second. Kafka offsets are a thing, consumer groups are a thing. It's trivial to ensure that only one message is delivered to only one consumer if that's what you want. Consumer groups track their offset and then commit the offset, the message stays in Kafka but it won't be read again.

This IMO is better behaviour than RabbitMQ since you can always re-read messages once they have been processed, whereas generally with MQ systems the message is then marked for deletion and asynchronously deleted.

> It's trivial to ensure that only one message is delivered to only one consumer

Exactly-once delivery is one of the hardest distributed systems problems. If you’ve “trivially” solved it, please show us your solution.

> It's trivial to ensure that only one message is delivered to only one consumer if that's what you want. Consumer groups track their offset and then commit the offset, the message stays in Kafka but it won't be read again. This IMO is better behaviour than RabbitMQ

The trivial solution is to use Kafka. They're clearly saying that Kafka makes it trivial, not that it's trivial to solve from scratch.

What the parent poster described isn’t what makes Kafka’s “exactly once” semantics work. It’s the use of an idempotency token associated with each publication, which effectively turns “at-least-once” semantics into effectively “exactly once” via deduplication.

We used Flink and S3 compatible backends (for the Flink state) to ensure exactly once processing.

I didn't say Kafka magically solves these problems for you, but it was required for the scalability we needed.

> better behaviour than RabbitMQ since you can always re-read messages once they have been processed

I can imagine, a 1 Billion dollar transaction accidentally gets processed by ten thousand client nodes due to a client app synchronization bug, company rethinks its dumb data dumper server strategy...news at 11.

It "disappears" in the sense that the Consumer-Group that read/committed that message (event) will never see it again. It doesn't "disappear" in the sense that a new Consumer-Group can be started in a way that will get that message, or you can reset your Consumer-Group's offset to re-consume it.

It is a *very bad* replacement for an MQ system, for the simple reason you can't quickly and effortlessly scale int/out consumers.

Why can't you? In my experience, scaling Kafka was far easier than scaling our RabbitMQ cluster. We started running into issues when our RabbitMQ cluster hit 25k TPS, our Kafka cluster of equivalent resources didn't break a sweat at 500k TPS.

Scaling consumers, not throughput. And it’s both directions (in and out, not just out).

What sort of systems do you work on to require this kind of traffic volume? I've worked on one project that I'd consider relatively high volume (UK Post Office Horizon Online) and we were only targeting 500 TPS.

Think extremely popular multiplayer videogames. We used these systems for all the backend infra, logins, logouts, chat messages, purchases.

We often had millions of players online at a given moment which means lots of transactions!

How did you handle re-partitioning and rebalancing every time you scaled your cluster in or out?

> you can make it work just as well as any MQ based system

you really can't. getting per-message acks, dynamically scaling competing consumers without having to repartition while retaining ordering, etc. requires a ton of hacks like client side tracking / building your own storage on top of offset metadata / etc.. and you still won't have all of the features actual message queues provide.

to make it worse, there is very little public work/discussion on this so you'll be on your own to figure out all of the quirks. the only notable example is https://github.com/confluentinc/parallel-consumer which is effectively abandoned

I don't think "while retaining ordering" is something you want to include here, since you can't guarantee processing order for any MQ system without serializing the consumption down to a single consumer.

absolute order, you're correct. but several systems support demultiplexing an interleaved stream by some key so that order is retained for all messages in each key space (pulsar key_shared subscriptions, azure service bus sessions, etc). you're still serializing consumption / have an exclusive consumer for each key, but with global parallelism.

here's the equivalent in parallel-consumer: https://github.com/confluentinc/parallel-consumer?tab=readme...

There is a whole KIP that is “preview” on Kafka 4.1 to handle this use case natively: https://cwiki.apache.org/confluence/plugins/servlet/mobile?c...

Note: I haven’t had a chance to test it out in anger personally yet.

yep i mentioned that in another thread below. very excited to see it added, but it will be some time until its production ready / they implement the full feature set

Don't you always need a database after reading events from Kafka to deduplication?

So the competing solutions are: PostgreSQL or Kafka+PostgreSQL

Kafka does provide some extrs there, handling load spikes, more clients that PG can handle natively and resilience to some DB downtime. But is it worth the complexity, in most cases no.

Actually you can avoid having a separate DB! You can build a materialized view of the data using [KTables](https://developer.confluent.io/courses/kafka-streams/ktable/) or use [interactive queries](https://developer.confluent.io/courses/kafka-streams/interac...). The "table" is built up from a backing kafka topic so you don't need maintain another datastore if the data view you want is entirely derived from one or more Kafka topics.

My understanding is this is pretty niche and can be complex.