Re:Mara: A lightweight ETL framework, halfway between plain ...
by martin_loetzsch on Wednesday May 09, 2018 @02:35PM
(#17032538)
(author here)
That's absolutely correct. Mara uses Python's multiprocessing [1] to parallelize pipeline execution [2] on a single node so it doesn't need a distributed task queue. Beyond that (and visualization) it can't do much. In fact it doesn't even have a scheduler (you can use jenkins or cron for that).
That's absolutely correct. Mara uses Python's multiprocessing [1] to parallelize pipeline execution [2] on a single node so it doesn't need a distributed task queue. Beyond that (and visualization) it can't do much. In fact it doesn't even have a scheduler (you can use jenkins or cron for that).
[1] https://docs.python.org/3.6/library/multiprocessing.html
[2] https://github.com/mara/data-integration/blob/master/data_in...