Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on Airbyte: Simple and extensible open-source EL(T). Parent comment
* Make sure you're only loading incremental updates
* Make sure you don't miss any data while doing incremental updates. This includes deleted rows.
* Update the Snowflake table schemas as Postgres schemas change. If this is impossible you need to alert someone.
* Keep historical metrics so you know if the job is slow or too much data or whatever.
Now do this for twenty different data sources every hour.
There are no comments on this comment yet.
* Make sure you're only loading incremental updates
* Make sure you don't miss any data while doing incremental updates. This includes deleted rows.
* Update the Snowflake table schemas as Postgres schemas change. If this is impossible you need to alert someone.
* Keep historical metrics so you know if the job is slow or too much data or whatever.
Now do this for twenty different data sources every hour.