News for hackers. Stuff that matters.

A comment on Ask HN: What is the simplest data orchestration tool you've worked with?. Parent comment

Re:Ask HN: What is the simplest data orchestration tool you'... by vector_spaces on Friday March 21, 2025 @04:17PM (#43440347)
Is there anything even more lightweight, where you don't have to write your code any differently? For instance, say I have 10 jobs that don't depend on each other, all of them pretty small.

Dagster and even Luigi feel like overkill but I'd still like to plug those into a unified interface where I can view previous runs, mainly logs and exit codes. Being able to do some light job configuration or add retries would be nice but not required. For the moment I just use a logging handler that writes to a database table and that's fine

[ Parent | on HN ]
Re:Ask HN: What is the simplest data orchestration tool you'... by cicdw on Friday March 21, 2025 @05:18PM (#43440917)
One of the goals of Prefect's SDK is to be minimally invasive from a code-standpoint (in the simplest case you only need two lines to convert a script to a `flow`). Our deployment model also makes infrastructure job config a first-class citizen so you might have a good time trying it out. (disclosure: work at Prefect)
[ Parent | on HN ]
Re:Ask HN: What is the simplest data orchestration tool you'... by aradox66 on Friday March 21, 2025 @08:51PM (#43442305)
Love prefect! but for workflows involving concurrency, Prefect code needs to get somewhat invasive.

Prefect relies on prefect.task()-wrapped methods as the lowest granularity of concurrency in a program, and requires you to use the (somewhat immature) prefect task APIs to implement that concurrency.

more on this complaint here: https://austinweisgrau.github.io/migrating-to-prefect-part-3...

[ Parent | on HN ]
Re:Ask HN: What is the simplest data orchestration tool you'... by cicdw on Friday March 21, 2025 @09:57PM (#43442599)
This is an excellent write up thank you for sharing! Yea, our concurrency API needs an upgrade - coincidentally this is going to be a theme of the next sprint or two so I hope I can report some improvements back soon.
[ Parent | on HN ]
Re:Ask HN: What is the simplest data orchestration tool you'... by disgruntledphd2 on Friday March 21, 2025 @04:21PM (#43440383)
I think that Airflow 2 implemented a decorator mode which you can just use on functions.

Honestly, just use airflow, it has its issues but it sucks in well known and predictable ways.

[ Parent | on HN ]