News for hackers. Stuff that matters.
Re:Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/S... by mattbessey on Tuesday January 26, 2021 @07:07PM (#25922571)
I had a read through your docs but was unable to find any info on how you handle the sync of hard deletions from sources.

We use Stitch at the moment and have found this to be a surprisingly hard problem to solve without binary log replication, and without full refreshes. For the moment we've ruled out binlogs as we use Aurora MySQL which requires you to binlog from the master, and tying my data warehouse replication to the master node concerns me.

In incremental mode, for common DBs e.g. MySQL / Postgres, will Airbyte pick up hard deletions ever?

Re:Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/S... by mtricot on Tuesday January 26, 2021 @07:45PM (#25922864)
We don’t today, we will tomorrow when we start working on CDC.

You’re raising a good point with MySQL, we will need to take this limitation into account. Hopefully there is a workaround.

Re:Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/S... by francoisp on Wednesday January 27, 2021 @10:55AM (#25929765)
check this out, https://github.com/francoisp/rosettable. Uses binlog a reader to call postgres triggers, upon which you can build 2 way sync realtime.
Re:Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/S... by francoisp on Wednesday January 27, 2021 @10:57AM (#25929790)
hello Matt, you might be interested in this: uses binlog to provide 2 way sync to postgres including hard delete detection, using postgres triggers. https://github.com/francoisp/rosettable