Have you done/seen any benchmarks between Redpanda/NATS and Kafka for this use case?

Some folks in SigNoz community have also suggested NATS for this, but I have not deep dived into benchmarks/features yet

Unfortunately no :/

Exactly-once semantics of what specifically? Or do you mean at-least-once ?

Exactly-once semantic between Kafka and the observability engine.

You have to do a bit more than that if you want exactly once end-to-end (I.E if Kafka itself can contain duplicates). One of my former colleagues did a good write up on how Husky does it: https://www.datadoghq.com/blog/engineering/husky-deep-dive/

Yeah, I was only talking about exactly once semantic between Kafka and Quickwit.

> exactly-once semantic without too much efforts: by keeping track of the positions of partitions in your own database and carefully acknowledging them when you are sure data is safely stored in your db

That's not really "exactly once". What happens when your system dies after it made sure the data is safely stored in the db and before ack-ing?

Depending on how you use the database it is. If you write the data as well as the offset to the DB in the same transaction, you can then seek to the offset stored in the DB after application restart and continue from there.

> after application restart and continue from there.

What if the application doesn't restart before the queue decides the message was lost and resends?

In Kafka the "queue" is dumb, it doesn't lose messages (it's an append only durable log) nor does it resend anything unless the consumer requests it.

There has to be a retry system somewhere, otherwise you'd end up with a 0-or-more delivery system if the app crashes after picking up from the queue, but never processing or ack-ing.

You should drop "(...) and carefully acknowledging them when you are sure data is safely stored in your db (...)" part then, because it means it's not necessary, you don't rely on it.

One-or-more semantics + local deduplication gives one-and-only semantics.

In this case you're optimising local deduplication with strictly monotonic index.

One downside is that you leak internals of other system (partitions).

The other is that it implies serialised processing - you can't process anything in parallel as you have single index threshold that defines what has been and what has yet not been processed.

I'm not the one who wrote the original comment, so I can't modify it. But one should still commit offsets because it is the happy-path; DB transaction successful? Commit offset. If the latter fails due to e.g application crash and you seek at startup to the partition offset stored in the DB + 1, you get exactly-once semantics. There's some more details, e.g you'd have to do the same during consumer group rebalance, and topic configuration also plays a role, for example if the topic is a compacted topic or not, and if you write tombstones, what its retention policy is.

Yes, it's good option but it requires serialised processing in partition scope, which may or may not be desirable.

If one-and-only-one semantics are needed and processing should be parallel, other methods have to be used.

Presumably every exactly once processing scenario needs you to squeeze things through a serial pipe at some point, or you could have 2 messages with the same ID come in and be processed in parallel?

Yes but scope/blocking/serialisation can be narrow or wide - ie. it can be per message id (highly parallel) or one for all messages of certain type/partition (not parallel).

Good point: first you're right, we do the ack on Kafka but it's not necessary. Second, this is not what I wanted to stress... and I should have not used the verb "acknowledge". What we do is upload the data on S3, then we commit partitions + positions in what we call the metastore. I can't edit my comment unfortunately.

> One downside is that you leak internals of other system (partitions).

True, but we generalized the concept of partitions for other datasources, pretty convenient to use it for distributing indexing tasks.

> The other is that it implies serialised processing - you can't process anything > in parallel as you have single index threshold that defines what has been and > what has yet not been processed.

Fortunately Kafka is partitioned. You cannot work in parallel along partitions.

Also, you can streamline your process. If you are running your data through operation (A, B, C). (C on batch N) can run at the same time as (B on batch N+1), and (A on batch N+2)

We do both at quickwit.

Isn't exactly once delivery the kind of problem like the CAP thereom where it's not possible?

You can make the downstream idemptoent wrt what the queue is delivering, but the queue might still redeliver things.