pg_flo – Stream, transform, and re-route PostgreSQL data in real-time

Re:pg_flo – Stream, transform, and re-route PostgreSQL data ... by 2000swebgeek on Monday November 04, 2024 @02:31AM (#42039366)

pg_flo looks like a very interesting project we might end up using. I like the archival to s3 and cheap versions of tools that get done at lower scale like GBs instead of TBs worth of data (Which is what I think Debezium does and something I can easily test locally)

The work/effort I need to put in with kafka etc., w/ Debezium is a short term effort, but evaluating the hassle.

Other solutions being evaluated by my team is peerdb - https://docs.peerdb.io/why-peerdb

[ Parent | on HN ]

Re:pg_flo – Stream, transform, and re-route PostgreSQL data ... by shayonj on Monday November 04, 2024 @03:03AM (#42039518)

Thanks for checking it out - Curious are you syncing/storing the data in some specific format in s3?

[ Parent | on HN ]

Re:pg_flo – Stream, transform, and re-route PostgreSQL data ... by mritchie712 on Monday November 04, 2024 @07:31AM (#42040948)

peerdb was acquired by clickhouse, so if you're syncing to a destination other than clickhouse, you may see support for it wane

[ Parent | on HN ]

Re:pg_flo – Stream, transform, and re-route PostgreSQL data ... by bzzzt on Monday November 04, 2024 @03:08PM (#42045529)

There's a standalone version of Debezium nowadays which can publish directly to a NATS queue. No Kafka required anymore.

[ Parent | on HN ]