News for hackers. Stuff that matters.
Re:Introduction to Apache Airflow by sdepablos on Friday May 29, 2020 @09:26AM (#23349707)
There're so many alternatives to Airflow nowadays that you really need to make sure that Airflow is the best solution (or even a solution) to your use case. There's plenty of use cases better resolved with tools like Prefect or Dagster, but I suppose the inertia to install the tool everyone knows about is really big.

BTW, here's https://github.com/pditommaso/awesome-pipeline a list of almost 200 pipeline toolkits.

Re:Introduction to Apache Airflow by walleeee on Friday May 29, 2020 @10:04AM (#23350075)
I've had a wonderful experience with Dagster so far. I love that it can deploy to Airflow, Celery, Dask, etc, I love the Dagit server and UI and that I can orchestrate pipelines over HTTP, I love the notebook integration via Papermill, I love that it's all free (looking at Prefect here...), and the team is extremely responsive on both Slack and GitHub
Re:Introduction to Apache Airflow by _frkl on Friday May 29, 2020 @10:16AM (#23350199)
Didn't Prefect open source their orchestration component recently, or am I mistaken? What part of Prefect is still closed?
Re:Introduction to Apache Airflow by walleeee on Friday May 29, 2020 @10:33AM (#23350350)
Oh, I was saying it wasn't free. I think you're right and it is fully open source
Re:Introduction to Apache Airflow by rywalker on Friday May 29, 2020 @12:46PM (#23352281)
It's not quite "open source", better labelled as "source available"
Re:Introduction to Apache Airflow by dmayle on Friday May 29, 2020 @09:56AM (#23350000)
I've gone through a large number of these, and I think that Airflow is the best on Kubernetes for managed orchestration. The things I like are:

  * Source control for workflows/DAGs (using git-sync)
  * Tracking/retries with SLAs
  * Jobs run in Kubernetes
  * Web UI for management
  * Fully open source
I also use Argo Workflows, because I like its native handling of Kubernetes objects (e.g. the ability to manage and update a deployment as one of the steps), but it just doesn't have the orchestration/tracking side of things very well managed yet
Re:Introduction to Apache Airflow by tiagod on Friday May 29, 2020 @10:20AM (#23350245)
I absolutely love Cadence/Temporal. It can do anything the others can do (and you could implement any other engine with it), without any DSLs.
Re:Introduction to Apache Airflow by bjacobso on Friday May 29, 2020 @12:23PM (#23351969)
I'd also like to +1 on Cadence/Temporal. It's a really great mental model and framework for thinking about workflows.

This is more of a proof of concept but it can also support DSLs (although we found the go client is easier to understand than DSLs): https://github.com/checkr/states-language-cadence

Re:Introduction to Apache Airflow by mfateev on Friday May 29, 2020 @01:07PM (#23352568)
As a team lead of the Cadence/Temporal it is really amazing to hear such a feedback. Thank you!

BTW. You can AMA about the technology here.

Re:Introduction to Apache Airflow by ttsda on Friday May 29, 2020 @01:30PM (#23352882)
Thank you for your work! I believe once it's properly documented and marketed, it will be very popular.
Re:Introduction to Apache Airflow by cpdean on Friday May 29, 2020 @10:47AM (#23350523)
I think this is the link to cadence: https://github.com/uber/cadence

and temporal: https://github.com/temporalio/temporal

Re:Introduction to Apache Airflow by tiagod on Friday May 29, 2020 @10:59AM (#23350683)
Yep, this is it. They're developed by the same team, Temporal is still beta but I believe the production version is coming out late-June.

I can't find an easy way to explain everything it does, but it pretty much allows you to write naive functions with no error handling, with month-long sleeps, auto-retries on unreliable function calls, etc etc.

It also gives you a web interface when you can inspect the running functions, and allows for external code (and other workflow functions) to signal/query the running workflows.

Re:Introduction to Apache Airflow by silasb on Friday May 29, 2020 @10:29AM (#23350321)
Just found out about Temporal and it looks interesting. I eager to jump in but our organization primarily uses Ruby. I know the big difference between Cadence vs Temporal is the fact they are using GRPC which seems much easier to adopt.
Re:Introduction to Apache Airflow by ttsda on Friday May 29, 2020 @11:00AM (#23350692)
I also find the API to be more consistent on Temporal (I began implementing a project in Cadence before finding out about Temporal)
Re:Introduction to Apache Airflow by mfateev on Friday May 29, 2020 @11:29AM (#23351107)
Temporal has an externally contributed Ruby client. It is going to be open sourced very soon.
Re:Introduction to Apache Airflow by silasb on Friday May 29, 2020 @11:44AM (#23351335)
Very exciting news. Is there a way that I can subscribe to get info on when it'll be released?
Re:Introduction to Apache Airflow by mfateev on Friday May 29, 2020 @12:00PM (#23351585)
There is #ruby-sdk channel in the Temporal Slack: https://join.slack.com/t/temporalio/shared_invite/zt-c1e99p8...
Re:Introduction to Apache Airflow by autogenerated on Friday May 29, 2020 @12:02PM (#23351618)
Moving off of Airflow and to Cadence/Temporal was the single biggest relief in terms of maintainability, operational ease and scalability. Also +1 on being free of any DSL.
Re:Introduction to Apache Airflow by ttsda on Friday May 29, 2020 @12:39PM (#23352167)
I'm currently moving from a custom yaml DSL-based engine to Temporal and it's the best architectural decision I've taken in a long time. I researched a lot and couldn't find anything that even came close to the freedom it provides.
Re:Introduction to Apache Airflow by alexg123 on Friday May 29, 2020 @07:49PM (#23356997)
Curious about this. Can you elaborate more? Also happy to hop on a call/zoom if you don't want to share publicly (email me at [email protected]). I'm working on something similar.
Re:Introduction to Apache Airflow by sequoia on Friday May 29, 2020 @01:08PM (#23352581)
> A curated list of awesome pipeline toolkits

Is this "curated"? It seems like an exhaustive "dump" of toolkits.

Re:Introduction to Apache Airflow by unixhero on Monday June 01, 2020 @11:31AM (#23379609)
What curation do you feel it lacks?
Re:Introduction to Apache Airflow by sequoia on Tuesday June 02, 2020 @01:22PM (#23393528)
To curate a collection is to be an editor, determining what to include and what to exclude. From a curated selection of tools, I expect to see a selection of the best tools, chosen by a knowledgable curator who has evaluated the tools in some way.

So for example, if you said "I want to start a library, donate any books you have at my house" that would not be a "curated" collection in my opionion. If you went through & evaluated the books, selecting only those that you'd personally recommend and discarding the rest, that would be a "curated" collection.

(Strictly speaking any maintenance of a collection even just "dump everything at my door at I'll put it on the pile" could be considered "curation" but to call a list of tools "curated" suggests there's some selection going on, and there does not appear to be on this list.)

EDIT: In fact, all of the definitions of "curate" here[0] start with the word "select," for example "Select, organize, and look after the items in (a collection or exhibition)." This fits my meaning. This definition[1] of "curator" requires one to merely have "care and superintendence of something," so in that sense the list is "curated" insofar as someone looks at PRs and clicks the "merge" button.

The Miriam-Webster definition, however, speaks of curating a more general "something" rather than a "collection." If you "curate" a statue by cleaning & protecting it, fine, you don't need to select the statue. However, when a collection is curated, in my opinion, this necessarily implies selection, not just maintenance.

0: https://www.lexico.com/en/definition/curate 1: https://www.merriam-webster.com/dictionary/curator

Re:Introduction to Apache Airflow by Peteris on Friday May 29, 2020 @09:32AM (#23349768)
One way to get around this is to use Kedro https://github.com/quantumblacklabs/kedro, which is the most minimal possible pipeline interface, yet allows you to export to other pipeline formats and/or build your own exporters.
Re:Introduction to Apache Airflow by sdepablos on Friday May 29, 2020 @09:37AM (#23349820)
Interesting (and in the list). Reading the documentation looks like the only other format it allows to be exported is Airflow, isn't it?
Re:Introduction to Apache Airflow by adolph on Friday May 29, 2020 @12:16PM (#23351860)
Here's a nice podcast about Prefect that talks about its and Airflow's approachs:

https://softwareengineeringdaily.com/2020/04/29/prefect-data...

Re:Introduction to Apache Airflow by eyeball on Friday May 29, 2020 @10:24AM (#23350277)
We used Luigi because airflow was to complicated to get an unsupportive IT department to install.