Re:Data Science at the Command Line, 2nd Edition (2021)
by fforflo on Monday May 06, 2024 @06:43AM
(#40273137)
Yes, that's extremely important. I've had great success, but replacing Airflow, Luigi, and friends with a cron-ed Makefile target refreshing some database tables (usually Materialized views).
I've then used this tool to visualize the execution graph. https://github.com/lindenb/makefile2graph
The result looks like this: https://tselai.com/data/graph.png
The convenient thing is that each node in the execution graph is in a different environment. Some are shell scripts, a some are Python scripts while others are SQL queries.
Re:Data Science at the Command Line, 2nd Edition (2021)
by dspillett on Monday May 06, 2024 @07:02AM
(#40273244)
There are also a few ways to spread make managed processes over multiple nodes, assuming they share storage, which could be useful if your transforms bottleneck at the CPU rather than on storage or network IO.
Re:Data Science at the Command Line, 2nd Edition (2021)
by daniel_grady on Monday May 06, 2024 @05:29PM
(#40279712)
The classics never go out of style. Mike Bostock has a nice article about the use of Make for data workflows: https://bost.ocks.org/mike/make/.
The sort of massive C++ build that make can handle are typically much more complicated than your average ETL pipeline. So, there is plenty of room to grow into make.