News for hackers. Stuff that matters.

A comment on Ask HN: What tools do you use for data munging and merging?.

Re:Ask HN: What tools do you use for data munging and merging? by westonsankey on Monday January 13, 2020 @10:40AM (#22035362)
I try to avoid GUI-based tools as they generally don't lend themselves well to version control, code reuse, etc. There are several orchestration tools that allow you to create arbitrarily complex ETL pipelines using SQL, Python, Spark, etc. Here are links to a few:

http://airflow.apache.org/

https://www.prefect.io/

https://dagster.readthedocs.io/en/0.6.7/

https://www.getdbt.com/

[ Parent | on HN ]
Re:Ask HN: What tools do you use for data munging and merging? by veritas3241 on Monday January 13, 2020 @11:06AM (#22035602)
dbt is a fantastic tool. We really on it for basically all of our transformations at GitLab.
[ Parent | on HN ]
Re:Ask HN: What tools do you use for data munging and merging? by pjot on Monday January 13, 2020 @01:49PM (#22037311)
+1 for dbt.

Great tool with a really active community along with it

[ Parent | on HN ]
Re:Ask HN: What tools do you use for data munging and merging? by eyeball on Monday January 13, 2020 @11:49PM (#22041933)
Dbt looks awesome. Too bad no oracle support.
[ Parent | on HN ]
Re:Ask HN: What tools do you use for data munging and merging? by drewbanin on Tuesday January 14, 2020 @01:00PM (#22047364)
Hi there - I run product for dbt - happy to chat more about Oracle support if you're interested. There are a handful of community-supported dbt plugins out in the wild. Oracle could very well be one of the next ones. Check some of these out to see what's involved in building a database plugin:

- https://github.com/fishtown-analytics/dbt-spark

- https://github.com/fishtown-analytics/dbt-presto

- https://github.com/mikaelene/dbt-sqlserver

- https://github.com/jacobm001/dbt-mssql

[ Parent | on HN ]
Re:Ask HN: What tools do you use for data munging and merging? by eyeball on Tuesday January 14, 2020 @08:08PM (#22050878)
cool. would be an interested user, but building a plug in ... beyond my team's ability.
[ Parent | on HN ]