Re:Antha – A high-level language for biology
by samuell on Friday December 05, 2014 @02:16PM
(#8706298)
I think flow based programming principles can do wonders for workflow systems and composable analytics pipelines.
We have recently used Spotify's luigi workflow system a lot, and ran into some tricky problems with a bit more complex workflows, that was only solved after we incorporated a bit of FBP principles (in/out ports and separate network definition, to be exact).
At least in my experience so far, FBP principles really help to create the kind of modularity and composability that is needed in these kind of applications.
We have recently used Spotify's luigi workflow system a lot, and ran into some tricky problems with a bit more complex workflows, that was only solved after we incorporated a bit of FBP principles (in/out ports and separate network definition, to be exact).
Have documented a bit of it here: https://medium.com/@saml/loosely-coupled-tasks-in-luigi-work... (fairly detailed and technical post though, not sure how easy it is to follow).
At least in my experience so far, FBP principles really help to create the kind of modularity and composability that is needed in these kind of applications.