Re:Airflow and the Future of Data Engineering: A Q&A
by tedmiston on Wednesday March 01, 2017 @12:55PM
(#13766279)
Thanks, this sounds really interesting. If this changes or you decide to write personal code with similar structure, I'd definitely like to discuss further. I'm also currently exploring DAG structure but with class and function abstractions with the goals of increasing code reuse and minimizing boilerplate around our custom operators.
Cookiecutter takes a Python code file as input with Jinja interspersed (the template). When you want to make a new instance from the template, it gives command-line prompts, and then evaluates the Jinja logic — custom variables, loops, etc to output Python code. It took me 15–30 minutes to get started and has already paid off.
Cookiecutter takes a Python code file as input with Jinja interspersed (the template). When you want to make a new instance from the template, it gives command-line prompts, and then evaluates the Jinja logic — custom variables, loops, etc to output Python code. It took me 15–30 minutes to get started and has already paid off.