Re:Stream Processing with DuckDB/Polars?
by alclol on Sunday December 22, 2024 @06:29PM
(#42490081)
Two ideas you might like:
Debezium + Arrow Flight: Use Debezium as a library to grab PostgreSQL CDC events and stream them into Arrow for super-fast, columnar processing. Works great with Polars or DuckDB.
RisingWave: This is a lightweight stream processor that connects directly to Postgres CDC, lets you write SQL for transformations, and keeps everything updated in real-time. No Kafka or heavy setups required.
Debezium + Arrow Flight: Use Debezium as a library to grab PostgreSQL CDC events and stream them into Arrow for super-fast, columnar processing. Works great with Polars or DuckDB.
RisingWave: This is a lightweight stream processor that connects directly to Postgres CDC, lets you write SQL for transformations, and keeps everything updated in real-time. No Kafka or heavy setups required.