westonsankey

Re:Ask HN: What tools do you use for data munging and merging? on Monday January 13, 2020 @10:40AM (#22035362)

I try to avoid GUI-based tools as they generally don't lend themselves well to version control, code reuse, etc. There are several orchestration tools that allow you to create arbitrarily complex ETL pipelines using SQL, Python, Spark, etc. Here are links to a few:

http://airflow.apache.org/

https://www.prefect.io/

https://dagster.readthedocs.io/en/0.6.7/

https://www.getdbt.com/

Strata NYC 2017 – Data Engineering Trends

Posted by westonsankey on Tuesday October 10, 2017 @09:18AM
from the medium.com dept.

https://medium.com/rv-data/strata-nyc-2017-and-the-state-of-data-engineering-155556b12d44

( Read More... | no comments | 2 points | on HN )

Apple's App Problem

Posted by westonsankey on Saturday February 09, 2013 @08:56PM
from the westonsankey.co dept.

http://westonsankey.co/blog/2013/2/9/apples-problem

( Read More... | no comments | on HN )