News for hackers. Stuff that matters.

A comment on Listen to Database Changes Through the Postgres WAL. Parent comment

Re:Listen to Database Changes Through the Postgres WAL by oulipo2 on Monday November 17, 2025 @04:09AM (#45952008)
Does it handle the things that the post mentions about the ever-growing WAL, and the fact that some listeners can go offline and need to get back old messages (eg if Kafka crashes?)
[ Parent | on HN ]
Re:Listen to Database Changes Through the Postgres WAL by gunnarmorling on Monday November 17, 2025 @04:33AM (#45952126)
Robustness is a key design goal of Debezium. It supports heart beating to address WAL growth issues (wrote about that issue at [1]). If Kafka crashes (or Debezium itself), it will resume consuming the replication slot from where it left off before (applying at-least once semantics, i.e. there can be duplicates in case of an unclean shut-down).

Naturally, if the consumer is down, WAL retained for that replication slot continues to grow until it comes back up again, hence monitoring is key (or the slot gets invalidated at a certain threshold, it will restart with a new initial snapshot).

Disclaimer: I used to lead the Debezium project

[1] https://www.morling.dev/blog/mastering-postgres-replication-...

[ Parent | on HN ]