News for hackers. Stuff that matters.
Re:ETL Pipelines with Airflow: The Good, the Bad and the Ugly by msinxi on Friday October 08, 2021 @06:38PM (#28804930)
Checkout great expectations (https://greatexpectations.io/). You basically run assertions on your data. It auto-generates documentation for you and you can also store results of your validations in a flat file or database. I think there are dbt modules that try to mimic great expectations' validation but I like the docs that come with great expectations
Re:ETL Pipelines with Airflow: The Good, the Bad and the Ugly by boxysean on Saturday October 09, 2021 @04:04AM (#28807931)
[dbt Labs employee here]

Check out dbt-expectations package[1]. It's a port of the Great Expectations checks to dbt as tests. The advantage of this is you don't need another tool for these pretty standard tests, and can be early incorporated into dbt workflows.

[1] https://github.com/calogica/dbt-expectations