News for hackers. Stuff that matters.
Re:Mara: A lightweight ETL framework, halfway between plain ... by abd12 on Wednesday May 09, 2018 @11:07AM (#17030389)
To me, it's easiest to think of it as "Make, for data". Your ETL pipelines are often a complex graph of dependencies. If some step halfway down the chain fails, you don't want to restart at the very beginning -- you want to restart where it failed and keep moving.

The ETL frameworks (Airflow, Luigi, now Mara) help with this, allowing you to build dependency graphs in code, determine which dependencies are already satisfied, and process those which are not. They'll usually contain helper code for common ETL tasks, such as interacting with a database, writing to/reading from S3, or running shell scripts.

Re:Mara: A lightweight ETL framework, halfway between plain ... by martin_loetzsch on Wednesday May 09, 2018 @02:08PM (#17032260)
(author here).

Mara data integration is indeed a glorified version of Make (with cost based scheduling and lots of visualizations)