Re:Mara: A lightweight ETL framework, halfway between plain ...
by artwr on Wednesday May 09, 2018 @12:19PM
(#17031061)
With a little bit of work you could probably trigger tasks on a Windows Machine, in particular if the work you need to do is mostly script and/or SQL based. The main limit is that Gunicorn for the Airflow webserver is not currently compatible with Windows, but the scheduler should work fine. I believe it might be able to run in the Windows Subsystem for Linux, but I don't think anyone has tested it as of yet. (Source: I am an Apache Airflow committer)
Re:Mara: A lightweight ETL framework, halfway between plain ...
by leblancfg on Wednesday May 09, 2018 @03:19PM
(#17032974)
Thanks! To be fair, I was looking at it from a single-user perspective, and was optimizing for:
time it takes to setup and move my ETL to Airflow + time I will save in the future by using the framework
vs time I will spend fixing and maintaining my already-existing ETL scripts
so I dropped the notion after I hit the first major snag. But I should put a reminder to take a look back at it in a few months!
Re:Mara: A lightweight ETL framework, halfway between plain ...
by stadeschuldt on Wednesday May 09, 2018 @11:54AM
(#17030807)
Mara is designed to be simpler than Airflow. It positions itself between plain scripts and Airflow.
Re:Mara: A lightweight ETL framework, halfway between plain ...
by huac on Wednesday May 09, 2018 @12:06PM
(#17030929)
What features does it forgo to be 'simpler'?
Re:Mara: A lightweight ETL framework, halfway between plain ...
by artwr on Wednesday May 09, 2018 @12:31PM
(#17031210)
The main feature which jumps out at me, I think it schedules on a single node, where Airflow can orchestrate across multiple nodes. Airflow is also starting to implement more enterprise-y features, like role-based access control, which should be coming to the next release.
Re:Mara: A lightweight ETL framework, halfway between plain ...
by martin_loetzsch on Wednesday May 09, 2018 @02:35PM
(#17032538)
(author here)
That's absolutely correct. Mara uses Python's multiprocessing [1] to parallelize pipeline execution [2] on a single node so it doesn't need a distributed task queue. Beyond that (and visualization) it can't do much. In fact it doesn't even have a scheduler (you can use jenkins or cron for that).
[1] https://docs.python.org/3.6/library/multiprocessing.html
[2] https://github.com/mara/data-integration/blob/master/data_in...
Would anyone care to explain how it differs from Airflow? I had dismissed Airflow some months ago for my use case (Windows, and large number of dependencies were an issue at the time), but would still like to eventually migrate my ETL scripts to a solid framework sometime in the future.