Issues we've encountered while building a Kafka based data processing pipeline

Re:Issues we've encountered while building a Kafka based dat... by dionian on Monday October 18, 2021 @10:36AM (#28905841)

Honest question, how do you expire content in Postgres? Every time I start to use it for ephemeral data I start to wonder if I should have used TimescaleDB or if I should be using something else...

[ Parent | on HN ]

Re:Issues we've encountered while building a Kafka based dat... by LoriP on Monday October 18, 2021 @11:32AM (#28906505)

You likely already know that TimescaleDB is an extension to PostgreSQL, so you get everything you'd get with PostgreSQL plus the added goodies of TimescaleDB. All that said, you can drop (or detach) data partitions in PostgreSQL (however you decide to partition...) Does that not do the trick for your use case, though? https://www.postgresql.org/docs/10/ddl-partitioning.html

Transparency: I work for Timescale

[ Parent | on HN ]

Re:Issues we've encountered while building a Kafka based dat... by chartpath on Thursday October 21, 2021 @07:30PM (#28952109)

Maybe something like https://github.com/citusdata/pg_cron or if it's not a rolling time window, just trigger functions that get called whenever some expression in SQL is true.

[ Parent | on HN ]