Re:Dabbling with Dagster vs. Airflow
by jmcnevin on Wednesday September 14, 2022 @02:36PM
(#32841352)
I'd say give Temporal (https://temporal.io) a look, but there are a lot of options (https://github.com/meirwah/awesome-workflow-engines).
Re:Dabbling with Dagster vs. Airflow
by skrtskrt on Wednesday September 14, 2022 @03:24PM
(#32842045)
Seconding Temporal, it is definitely a second-gen workflow platform with approaches that make many Airflow frustrations nonexistent.
In particular, workflows are extremely durable, they can easily just sit indefinitely waiting on a condition and you can redeploy workers whenever, and workflows will resume exactly where they were.
Re:Dabbling with Dagster vs. Airflow
by valenterry on Wednesday September 14, 2022 @04:11PM
(#32842713)
Wow, that looks cool. Especially since the way of defining things in code looks pretty first-class.
Shameless self-promotion: I built Eventline (see my profile) precisely because I just wanted to schedule and run code in various environments, manually or in reaction to various events, without the complexity that goes with specialized tools.
It gives you a pretty flexible job system, identities (secrets), monitoring, notifications, a web interface, a command line tool and a HTTP API. You can build lots of things above it.
Do you have specific use cases in mind ?
AWS Simple Workflows or Azure Logic Apps are both services that let you define S2S workflows however you like without any particular bias to CI/CD or business operations.
If you want to go even lower level, a framework like DTFx lets you define long-running, distributed and resilient orchestrations in code:
Re:Dabbling with Dagster vs. Airflow
by samhita-alla on Thursday September 15, 2022 @07:22AM
(#32850105)
Have you seen https://github.com/flyteorg/flyte? It's a data, ML and infrastructure orchestrator. (btw, I'm a Flyte engineer)
Re:Dabbling with Dagster vs. Airflow
by valenterry on Thursday September 15, 2022 @11:04AM
(#32853098)
Bonus points for having a Scala SDK!
Re:Dabbling with Dagster vs. Airflow
by lemmonii on Wednesday September 14, 2022 @01:27PM
(#32840368)
Argo Workflows
Re:Dabbling with Dagster vs. Airflow
by valenterry on Wednesday September 14, 2022 @01:42PM
(#32840603)
More like a SaaS that I can use without setting up my own infrastructure. I guess one could turn Argo Workflows into such an SaaS.
1.) data pipelines / data-science
2.) cicd / build pipelines
3.) ... you name it
I mean, just a service that gives me all the groundwork to build one of the above myself.
Is there something like that?