Mara: A lightweight ETL framework, halfway between plain scripts and Airflow

Re:Mara: A lightweight ETL framework, halfway between plain ... by endlessvoid94 on Wednesday May 09, 2018 @01:10PM (#17031605)

In my experience one of the places where every framework breaks down is when you must combine / reconcile multiple rows from multiple data sources to produce one row in a fact table.

Does there exist a "framework" that lets me do this simply?

[ Parent | on HN ]

Re:Mara: A lightweight ETL framework, halfway between plain ... by martin_loetzsch on Wednesday May 09, 2018 @02:42PM (#17032615)

(author here)

The mara example project [1] does exactly that. It combines PuPI download stats with Github repo activity data.

[1] https://github.com/mara/mara-example-project

[ Parent | on HN ]

Re:Mara: A lightweight ETL framework, halfway between plain ... by endlessvoid94 on Wednesday May 09, 2018 @05:44PM (#17034440)

Thanks! Just took a look.

The file directory structure is a bit confusing -- could you point me to the file that performs this transformation?

[ Parent | on HN ]

Re:Mara: A lightweight ETL framework, halfway between plain ... by martin_loetzsch on Wednesday May 09, 2018 @06:26PM (#17034774)

For example the PyPI download stats pipeline is here: https://github.com/mara/mara-example-project/tree/master/app...

The __init__.py contains the pipeline, and the rest is the SQL files that do the transformations

[ Parent | on HN ]

Re:Mara: A lightweight ETL framework, halfway between plain ... by endlessvoid94 on Wednesday May 09, 2018 @09:10PM (#17035719)

Thank you!

[ Parent | on HN ]