Re:Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/S...
by mtricot on Tuesday January 26, 2021 @07:45PM
(#25922864)
We don’t today, we will tomorrow when we start working on CDC.
You’re raising a good point with MySQL, we will need to take this limitation into account. Hopefully there is a workaround.
Re:Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/S...
by francoisp on Wednesday January 27, 2021 @10:55AM
(#25929765)
check this out, https://github.com/francoisp/rosettable. Uses binlog a reader to call postgres triggers, upon which you can build 2 way sync realtime.
Re:Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/S...
by francoisp on Wednesday January 27, 2021 @10:57AM
(#25929790)
hello Matt, you might be interested in this: uses binlog to provide 2 way sync to postgres including hard delete detection, using postgres triggers.
https://github.com/francoisp/rosettable
We use Stitch at the moment and have found this to be a surprisingly hard problem to solve without binary log replication, and without full refreshes. For the moment we've ruled out binlogs as we use Aurora MySQL which requires you to binlog from the master, and tying my data warehouse replication to the master node concerns me.
In incremental mode, for common DBs e.g. MySQL / Postgres, will Airbyte pick up hard deletions ever?