News for hackers. Stuff that matters.
Re:Data Science at the Command Line, 2nd Edition (2021) by fforflo on Monday May 06, 2024 @06:43AM (#40273137)
Yes, that's extremely important. I've had great success, but replacing Airflow, Luigi, and friends with a cron-ed Makefile target refreshing some database tables (usually Materialized views).

I've then used this tool to visualize the execution graph. https://github.com/lindenb/makefile2graph

The result looks like this: https://tselai.com/data/graph.png

The convenient thing is that each node in the execution graph is in a different environment. Some are shell scripts, a some are Python scripts while others are SQL queries.

There are no comments on this comment yet.