News for hackers. Stuff that matters.
Re:ETL Pipelines with Airflow: The Good, the Bad and the Ugly by DevKoala on Friday October 08, 2021 @03:58PM (#28803489)
My team is currently evaluating Debezium for a PG to S3 CDC solution, and is very advanced with the PoC.

What would be the argument for trying AirByte? It is the first time we hear about it.

Re:ETL Pipelines with Airflow: The Good, the Bad and the Ugly by Arimbr on Friday October 08, 2021 @04:40PM (#28803930)
Airbyte CDC is based on Debezium, but Airbyte abstracts it away and make it easier to CDC from Postgres, MySQL, MSSQL to any supported destination (included S3). Here is the doc for CDC: https://docs.airbyte.io/understanding-airbyte/cdc

I guess one benefit is that you can use Airbyte for all your data syncs, CDC and non-CDC. You can give it a try with your own data, and see if it's easier for your team. You can run Airbyte locally with Docker Compose: https://docs.airbyte.io/quickstart/deploy-airbyte

Re:ETL Pipelines with Airflow: The Good, the Bad and the Ugly by gunnarmorling on Friday October 08, 2021 @05:11PM (#28804259)
Can you elaborate on how Airbyte makes things easier for a user? Would love to pick up any potential improvements in Debezium itself, so that all its users get to benefit from them, rather than only users of a specific integrator like Airbyte.

Disclaimer: I work on Debezium