Re:Airbnb Engineering
by chuckcode on Thursday June 04, 2015 @06:45PM
(#9662701)
Seems like it is getting to be a trend to write your own workflow engine and open source it. Spotify has luigi, LinkedIn has Azkaban and Airbnb has Airflow. I wonder at what market capitalization companies start rolling their own language like .net, go and swift?
[ Parent
| on HN ]
How is that in any way surprising? The more resources you have, the more custom-tailored you can make your tools. And I can't imagine it's a shock that data workflows aren't a completely-solved-for-all-usecases problem at this point...
Don't forget Chronos.
Re:Airbnb Engineering
by toomuchtodo on Thursday June 04, 2015 @09:12PM
(#9663288)
Isn't Chronos more of a distributed scheduler ("fault tolerant cron") than a workflow engine?
Distributed job scheduler is, in a larger sense, a workflow engine. It will need to understand what is the next job to trigger, handle different situation where job fails\timeouts and do the retry if specified.
which was also started at Airbnb.
drake[0] from factual falls into this category as well.
Airflow reminded me a lot of taskflow from the OpenStack project.
Well, Facebook spun PHP off into Hack if that counts.
All a small framework called 'Ruby on Rails', released by Basecamp.
Re:Airbnb Engineering
by kroosaidher on Friday June 05, 2015 @12:21AM
(#9663885)
Hey, isn't a DSL your own language, strictly speaking? So any internal company-specific XML schema etc...