Re:Airflow's Problem
by zentrus on Tuesday August 02, 2022 @04:23PM
(#32323831)
Airflow helped my team out a lot a couple years ago mainly for the simplicity of the topdown UI-based view of a complicated ETL AND the ability to retry parts of the ETL.
We had lots of lessons learned. For instance, why does PythonOperator even exist? It takes a callable and thus you're likely not going to see good coding pattern emerge for something that needs to be 1000+ LoC. Instead, we just subclassed BaseOperator and used tried-and-true OO principles.
We had lots of lessons learned. For instance, why does PythonOperator even exist? It takes a callable and thus you're likely not going to see good coding pattern emerge for something that needs to be 1000+ LoC. Instead, we just subclassed BaseOperator and used tried-and-true OO principles.