The Wonders of Postgres Logical Decoding Messages

Re:The Wonders of Postgres Logical Decoding Messages by gunnarmorling on Sunday April 02, 2023 @02:38AM (#35407803)

Debezium relies on Kafka Connect's `OffsetBackingStore` contract [1] for persisting and restoring offset positions (that's subject to abstraction from Connect's SPIs at some point). While acknowledged LSNs are committed to the source DB in case of Postgres indeed, the offset store is still needed in most cases, e.g. for persisting other information, e.g. the fact whether or not an initial snapshot has happened yet.

> I don’t want to manage that in data producers anymore.

Could you elaborate a bit? The outbox pattern typically _is_ implemented in source applications, so I'm curious what alternative you have in mind? Stream processing e.g. with Flink actually can be an alternative, creating external data representations after the fact, though challenges arise e.g. in terms of transactional consistency.

[1] https://debezium.io/documentation/reference/stable/developme...

[ Parent | on HN ]

Re:The Wonders of Postgres Logical Decoding Messages by aynyc on Sunday April 02, 2023 @07:49AM (#35409567)

Our current many outbox patterns are implemented by producers using two phase commit. Rather than writing code for that, I want move the outbox to another component, but we don't want to use stream, we want to use a queue. Essentially, listen to WAL, any relevant data gets written to a queue for down stream consumers. Kafka/Kinesis is not in our stack right now, so I want to use standalone Debezium server on a AWS Fargate/ECS service and write to FIFO-SNS.

[ Parent | on HN ]