News for hackers. Stuff that matters.
Re:Data Science at the Command Line, 2nd Edition (2021) by usgroup on Monday May 06, 2024 @03:57AM (#40272279)
Worth highlighting that make defines a DAG, and that you can run make tasks in parallel which will automatically bottleneck (as desired) on common dependencies, and fan out otherwise.

The sort of massive C++ build that make can handle are typically much more complicated than your average ETL pipeline. So, there is plenty of room to grow into make.

Re:Data Science at the Command Line, 2nd Edition (2021) by fforflo on Monday May 06, 2024 @06:43AM (#40273137)
Yes, that's extremely important. I've had great success, but replacing Airflow, Luigi, and friends with a cron-ed Makefile target refreshing some database tables (usually Materialized views).

I've then used this tool to visualize the execution graph. https://github.com/lindenb/makefile2graph

The result looks like this: https://tselai.com/data/graph.png

The convenient thing is that each node in the execution graph is in a different environment. Some are shell scripts, a some are Python scripts while others are SQL queries.

Re:Data Science at the Command Line, 2nd Edition (2021) by dspillett on Monday May 06, 2024 @07:02AM (#40273244)
There are also a few ways to spread make managed processes over multiple nodes, assuming they share storage, which could be useful if your transforms bottleneck at the CPU rather than on storage or network IO.
Re:Data Science at the Command Line, 2nd Edition (2021) by daniel_grady on Monday May 06, 2024 @05:29PM (#40279712)
The classics never go out of style. Mike Bostock has a nice article about the use of Make for data workflows: https://bost.ocks.org/mike/make/.