News for hackers. Stuff that matters.
Re:Mara: A lightweight ETL framework, halfway between plain ... by stadeschuldt on Wednesday May 09, 2018 @11:54AM (#17030807)
Mara is designed to be simpler than Airflow. It positions itself between plain scripts and Airflow.
Re:Mara: A lightweight ETL framework, halfway between plain ... by huac on Wednesday May 09, 2018 @12:06PM (#17030929)
What features does it forgo to be 'simpler'?
Re:Mara: A lightweight ETL framework, halfway between plain ... by artwr on Wednesday May 09, 2018 @12:31PM (#17031210)
The main feature which jumps out at me, I think it schedules on a single node, where Airflow can orchestrate across multiple nodes. Airflow is also starting to implement more enterprise-y features, like role-based access control, which should be coming to the next release.
Re:Mara: A lightweight ETL framework, halfway between plain ... by martin_loetzsch on Wednesday May 09, 2018 @02:35PM (#17032538)
(author here)

That's absolutely correct. Mara uses Python's multiprocessing [1] to parallelize pipeline execution [2] on a single node so it doesn't need a distributed task queue. Beyond that (and visualization) it can't do much. In fact it doesn't even have a scheduler (you can use jenkins or cron for that).

[1] https://docs.python.org/3.6/library/multiprocessing.html

[2] https://github.com/mara/data-integration/blob/master/data_in...