See this Dec 2019 presentation by Pulsar committers, where they explain all this in more detail, i.e., the lack of transactions and the resulting limitations, and the motivation for adding such transactions to Pulsar. The approach looks very similar to Kafka's. https://www.slideshare.net/streamnative/transaction-preview-... The original ETA for transactions was Pulsar v2.6 (June 2020), but as of today there's still quite some work to be done (https://github.com/apache/pulsar/issues/2664). The latest ETA seems to be around the end of the year.
The key difference for an end user is that Kafka released all the functionality in one go back in 2017 (idempotent producer, transactions; which fwiw also explains why designing+building+testing took the Kafka community that long) so it has been much easier to understand what is actually supported vs. what is not.
For example, with Kafka Streams, any app you build with it just needs to set “processing.guarantee” to “exactly_once” in its configuration, and regardless of what happens to the app or its environment it will not lose messages (on write) or miss messages (on read) from Kafka.
Consider asking your question with a few more details in the Kafka user mailing list [1], or in the Confluent Community Slack [2] if you prefer chatting.
[1] https://kafka.apache.org/contact [2] https://launchpass.com/confluentcommunity
I use Kafka and needed a cache based on incoming events which were partitioned.
But if the consumer crashes it's not easy to pick up from exactly where it left off without manually committing offsets which hurts performance. There's also some hand waving Kafka gossip that it's hard to commit offsets right. Further suppose a task consumes events from topic A partition 3, produces events to topic B. Again, if the task crashes it's not clear what needs replaying in order to not to lose messages (on write) or missing messages (on read).
Any insight here is greatly appreciated.