Re:Apache Hop 2.0
by herodoturtle on Thursday June 09, 2022 @03:03AM
(#31678328)
This was a really useful comment. Thank you.
I know there's a degree of oversimplification going on here, but there's something to be said for having a simple bullet-list breakdown of all the use-cases - alongside the best tool for each use-case.
It is servers as a practical starting point in terms of narrowing down the list of tools (of which there are so many), before one proceeds with a deeper dive into the best fitting tool.
Would be great if there were a site that did this sort of thing for all the common architectural needs.
+1 to AWS Step Functions, in my last three companies I have built fairly complicated workflows with them and once you get used to them they are very powerful, reliable and cheap.
I just wish a little bit more monitoring on top of them but nothing you can not build by yourself.
Re:Apache Hop 2.0
by otabdeveloper4 on Thursday June 09, 2022 @06:24AM
(#31679496)
I ended up choosing NodeRed at current job. What does that make me? :)
"My team already knows Airflow and/or I want to pay Astronomer a lot of money" -> Airflow
"I love YAML and everything is on k8s anyway" -> Argo
"I just want something that works out of the box and don't want to host my own compute" -> Shipyard, maybe Orchest
"I want a more flexible, generic workflow engine and don't care about writing orchestration in Python" -> Temporal/Cadence
"I am very nostalgic" -> Azkaban, Oozie, Luigi
"I love clunky Java solutions to data problems" -> Nifi et al
"I like to pay for half-managed solutions and late upgrades to a first-generation technology" -> AWS/GCP hosted Airflow options
"I am on AWS and it doesn't need to be complicated" -> AWS Step Functions