Re:ETL Pipelines with Airflow: The Good, the Bad and the Ugly
by dominotw on Friday October 08, 2021 @11:34PM
(#28806813)
>the answer is usually "use this airflow operator to dump it into GCS and then use this airflow operator to load it into BigQuery"
why not hide these steps behind a yaml file and construct dags from those. devs already are used to writing yaml files for ci, kubernetes ect.
Re:ETL Pipelines with Airflow: The Good, the Bad and the Ugly
by Arimbr on Friday October 08, 2021 @11:44AM
(#28800513)
Nice work there! I also think that the next challenge for data teams is all this data documentation and discovery work.
I still think that Airflow is great for power data engineers. Airbyte and dbt are positioned to empower data analysts (or lazy data engineers like me) to own the ELTs.
Re:ETL Pipelines with Airflow: The Good, the Bad and the Ugly
by rockostrich on Friday October 08, 2021 @12:27PM
(#28801097)
Agreed. I see a lot of folks coming up with one off solutions for pulling data out of 3rd party sources like Kustomer or Lever. Giving a centralized UI for setting that up would be a great service.
Seems like I have a fun weekend project.
A mesh is certainly something in-between a lake and a warehouse... Something super simple that I've gotten good feedback on so far from DSs is just documenting the transformed data in place. It was really difficult to do this in our old ETL stack (data pulled from HBase, transformed to parquet + Hive in HDFS) but we've moved a lot of it over to Avro files loaded into BigQuery where we can just put decorators on our Scala transformation code that's writing the Avro files and that updates the schema with descriptions in BigQuery. Gives a nice bit of ownership to the engineering team and lets the DS using the data be a lot more autonomous. That boundary has to exist somewhere (or I guess in many places for a "mesh") so having it distinctly at data getting loaded in feels right to me.