News for hackers. Stuff that matters.
Re:Dabbling with Dagster vs. Airflow by valenterry on Wednesday September 14, 2022 @01:06PM (#32840065)
Where is a service that doesn't focus specifically on

1.) data pipelines / data-science

2.) cicd / build pipelines

3.) ... you name it

I mean, just a service that gives me all the groundwork to build one of the above myself.

Is there something like that?

Re:Dabbling with Dagster vs. Airflow by jmcnevin on Wednesday September 14, 2022 @02:36PM (#32841352)
I'd say give Temporal (https://temporal.io) a look, but there are a lot of options (https://github.com/meirwah/awesome-workflow-engines).
Re:Dabbling with Dagster vs. Airflow by skrtskrt on Wednesday September 14, 2022 @03:24PM (#32842045)
Seconding Temporal, it is definitely a second-gen workflow platform with approaches that make many Airflow frustrations nonexistent.

In particular, workflows are extremely durable, they can easily just sit indefinitely waiting on a condition and you can redeploy workers whenever, and workflows will resume exactly where they were.

Re:Dabbling with Dagster vs. Airflow by valenterry on Wednesday September 14, 2022 @04:11PM (#32842713)
Wow, that looks cool. Especially since the way of defining things in code looks pretty first-class.
Re:Dabbling with Dagster vs. Airflow by galdor on Wednesday September 14, 2022 @01:15PM (#32840192)
Shameless self-promotion: I built Eventline (see my profile) precisely because I just wanted to schedule and run code in various environments, manually or in reaction to various events, without the complexity that goes with specialized tools.

It gives you a pretty flexible job system, identities (secrets), monitoring, notifications, a web interface, a command line tool and a HTTP API. You can build lots of things above it.

Do you have specific use cases in mind ?

Re:Dabbling with Dagster vs. Airflow by rhplus on Wednesday September 14, 2022 @02:09PM (#32840934)
AWS Simple Workflows or Azure Logic Apps are both services that let you define S2S workflows however you like without any particular bias to CI/CD or business operations.

If you want to go even lower level, a framework like DTFx lets you define long-running, distributed and resilient orchestrations in code:

https://github.com/Azure/durabletask

Re:Dabbling with Dagster vs. Airflow by samhita-alla on Thursday September 15, 2022 @07:22AM (#32850105)
Have you seen https://github.com/flyteorg/flyte? It's a data, ML and infrastructure orchestrator. (btw, I'm a Flyte engineer)
Re:Dabbling with Dagster vs. Airflow by valenterry on Thursday September 15, 2022 @11:04AM (#32853098)
Bonus points for having a Scala SDK!
Re:Dabbling with Dagster vs. Airflow by lemmonii on Wednesday September 14, 2022 @01:27PM (#32840368)
Argo Workflows
Re:Dabbling with Dagster vs. Airflow by valenterry on Wednesday September 14, 2022 @01:42PM (#32840603)
More like a SaaS that I can use without setting up my own infrastructure. I guess one could turn Argo Workflows into such an SaaS.