News for hackers. Stuff that matters.
Re:Lessons learned from running Apache Airflow at scale by arinlen on Monday May 23, 2022 @01:04PM (#31481698)
> As mentioned elsewhere, AWS step functions are really the best in orchestration.

AWS Step Functions is a proprietary service provided exclusively by AWS, which reacts to events from AWS services and calls AWS Lambdas.

Unless you're already neck-deep in AWS, and are already comfortable paying through the nose for trivial things you can run yourself for free, it's hardly appropriate to even bring up AWS Step Functions as a valid alternative. For instance, Shopify's articles explicitly mention they are running their services in Google Cloud. Would it be appropriate to tell them to just migrate their whole services to AWS just because you like AWS Step Functions?

Re:Lessons learned from running Apache Airflow at scale by riku_iki on Monday May 23, 2022 @01:31PM (#31482105)
> already comfortable paying through the nose for trivial things you can run yourself for free

But fault tolerant workflow engine is not trivial thing, it may cost you many engineer hours to build, monitor and maintain it, so outsourcing it to someone else is totally viable solution.

Re:Lessons learned from running Apache Airflow at scale by arinlen on Monday May 23, 2022 @02:09PM (#31482775)
> But fault tolerant workflow engine is not trivial thing,

The complexity and risk of migrating cloud providers eclipses whatever problem you assign to "fault tolerant workflow engines".

Any mention of AWS Step Functions makes absolutely no sense at all and reads at best like a non-sequitur.

Re:Lessons learned from running Apache Airflow at scale by thorum on Monday May 23, 2022 @03:37PM (#31483916)
I read it as a comment on the UX / developer experience, which can superior with Step Functions vs competition regardless of whether Step Functions is an appropriate (or even physically possible) option for non-AWS projects.
Re:Lessons learned from running Apache Airflow at scale by Jugurtha on Monday May 23, 2022 @02:31PM (#31483137)
That was one the reasons we do "bring your own compute" with https://iko.ai so people who already have a billing account on AWS, GCP, Azure, DigitalOcean, can just get the config for their Kubernetes clusters and link them to iko.ai and their machine learning workloads will run on whichever cluster they select.

If you get a good deal from one cloud provider, you can get started quickly.

It's useful even for individuals such as students who get free credits from these providers: create a cluster and you're up and running in no time.

Our rationale was that we didn't wanted to be tied to one cloud provider.

Re:Lessons learned from running Apache Airflow at scale by parsnips on Monday May 23, 2022 @01:17PM (#31481898)
https://github.com/checkr/states-language-cadence allows you to define workflows in states language over cadence.
Re:Lessons learned from running Apache Airflow at scale by literallyWTF on Monday May 23, 2022 @01:13PM (#31481838)
This is another symptom of a person who doesn’t know what they’re talking about really.

It’s like those stackoverflow answers that tell the user to stop using PHP and rewrite it in Python or something.