Re:Flow-based Programming
by jpaulmorrison on Saturday January 10, 2015 @06:22PM
(#8868333)
There's a good discussion in the Google group of the ideas that differentiate what we are now calling "classical" FBP (or CFBP) from FBP-like systems, more and more of which are appearing all the time - in particular the topic "Underlying models" - https://groups.google.com/forum/#!topic/flow-based-programmi... .
If you are interested in the more large volume, data-processing, "classical" FBP implementations, a good place to start is my book, "Flow-Based Programming", 2nd ed., and the web site http://www.jpaulmorrison.com/fbp/ .
Re:Flow-based Programming
by SOMMS on Wednesday January 14, 2015 @02:39PM
(#8888507)
Hey Paul,
Glad to hear FBP is going strong. You may not remember me but I had a chance to work with you many years ago on getting the Java based FBP framework to input/output HTML (ie like a servlet node). Always loved the paradigm, the parallelism and genericness of the whole concept. I wrote many, many networks and eventually wrote an FBP parser that would create programatic FBP networks from XML files. I was wondering if this sort of thing (interpreter/parser/language) was ever incorporated into FBP itself? If not - are their any plans? Nowadays JSON would be the way to go. Thanks again - hope all is well. Chris S
Re:Flow-based Programming
by tbirdz on Saturday January 10, 2015 @04:48PM
(#8868058)
On the topic of dataflow programming I found the book "Dataflow and Reactive Programming Systems: A Practical Guide to Developing Dataflow and Reactive Programming Systems" by Matt Carkci [0] to give a pretty good overview of the different types and implementations of dataflow programming systems. I think that flow-based programming sounds incredibly cool, but I haven't found any project where I could use it well... yet!
Re:Flow-based Programming
by amelius on Saturday January 10, 2015 @04:59PM
(#8868092)
I think the biggest problems with flow based or reactive programming are the occurrence of glitches, and, more importantly, the problem of how to compute updates without performing unnecessary work.
I wonder if these problems have been successfully addressed somewhere.
Re:Flow-based Programming
by seanmcdirmid on Saturday January 10, 2015 @07:58PM
(#8868610)
My work [1] addresses glitches head on (hence the system is called "Glitch") by tolerating glitches and at least making the extra work functionally benign (via logging and rollback, like a transaction). Glitch is not based on data flow, however, but somewhere in between even if still very reactive.
I've built a few data flow systems in the past (my dissertation work [2] was based on one), but I decided to tolerate rather than avoid glitches early on because I found that way to be more resilient. Flapjax [3] avoids glitching via a topological sort, which I rather see as an optimization (to reduce unnecessary work) rather than as a correctness issue.
Re:Flow-based Programming
by amelius on Saturday January 10, 2015 @08:09PM
(#8868633)
Interesting. Do you know of any work that computes a data flow graph as the program runs (i.e. data flow is implicit in the source code), which performs then reactively?
Re:Flow-based Programming
by seanmcdirmid on Saturday January 10, 2015 @08:15PM
(#8868649)
Ya, that is exactly how Glitch works, except I just call the data flow graph as a dependency graph :).
Computing the dependency graph dynamically obviously leads to a lot more flexibility. I'm not really sure what you mean by "reactively" here: if you mean does it reactively update computations as state (and even code, if you want to get Bret Victor about it) changes, then ya.
Data flow by itself isn't necessarily reactive; actually the way it was originally defined to work very lazy push things through. Continuous interpretations where changes were propagated non-lazily didn't come until much later.
Re:Flow-based Programming
by harlowja on Saturday January 10, 2015 @06:21PM
(#8868331)
A framework that is similar to this in python (that I am one of the authors of) [1] (and [2] for more in-depth docs/examples). It's used in openstack (which some people may have heard of) where appropriate (and where desirable...) to help in consistent, scalable execution using a flow like methodology to declaratively describe workflows (and execute them using various strategies/engines).
Re:Flow-based Programming
by SEJeff on Sunday January 11, 2015 @10:26AM
(#8870172)
Just gonna say that task flow is wonderful software that is usable outside of the giant openstack ecosystem. I use it for some internal projects and am quite pleased with it.
Re:Flow-based Programming
by i000 on Sunday January 11, 2015 @11:13AM
(#8870289)
Author of PaPy here.
Thanks for giving PaPy a try. Although PaPy [https://github.com/mcieslik-mctp/papy] has never gained traction :(, I have been using it daily for over 5 years (no bugs in the scheduler so far). By now our PaPy based genomics pipelines have probably processed petabytes of data.
Re:Flow-based Programming
by samuell on Saturday January 10, 2015 @04:48PM
(#8868061)
Matt Carkci's book discusses the different closely related topics of data flow, flow based programming and reactive programming, very clearly and i quite some detail:
Re:Flow-based Programming
by mattcarkci on Sunday January 11, 2015 @10:07PM
(#8872326)
Matt Carkci here (the book author)...
Thank you for the kind words... everyone.
When I started writing the book "Dataflow and Reactive Programming Systems" I thought I completely understood the various forms of dataflow. Yet in doing research for the book I found there are many more subtle variations than I ever knew existed.
The book went through many iterations before I felt I accurately described the similarities and differences between the many forms of dataflow. I'm happy to hear that I may have succeeded.
Re:Flow-based Programming
by tbirdz on Saturday January 10, 2015 @04:49PM
(#8868066)
Wow, we just recommended the same book within seconds of each other. But seriously it's a good book, if you have interest in the topic it's worth reading.
Re:Flow-based Programming
by seanmcdirmid on Saturday January 10, 2015 @08:12PM
(#8868642)
Not really related. FRP has as much in common with dataflow programming as generic functional programming does. Related incidentally in that both refuse side effects, but one is based on function application while the other is based on data flow connections.
I'm not really familiar with how flow-based programming differs from data flow programming, however.
Re:Flow-based Programming
by whitten on Tuesday January 20, 2015 @06:42PM
(#8920508)
I can testify to the usefulness of Mozart Oz. It is a multi-paradigm programming system. I didn't know it had dataflow as one of its paradigms. Is this something written in Mozart Oz rather than an enhancement to it ?
Re:Flow-based Programming
by ColinWright on Sunday January 11, 2015 @04:17AM
(#8869517)
Bandwidth Limit Exceeded
The server is temporarily unable to service
your request due to the site owner reaching
his/her bandwidth limit.
Please try again later.
If you are interested in the more large volume, data-processing, "classical" FBP implementations, a good place to start is my book, "Flow-Based Programming", 2nd ed., and the web site http://www.jpaulmorrison.com/fbp/ .
Glad to hear FBP is going strong. You may not remember me but I had a chance to work with you many years ago on getting the Java based FBP framework to input/output HTML (ie like a servlet node). Always loved the paradigm, the parallelism and genericness of the whole concept. I wrote many, many networks and eventually wrote an FBP parser that would create programatic FBP networks from XML files. I was wondering if this sort of thing (interpreter/parser/language) was ever incorporated into FBP itself? If not - are their any plans? Nowadays JSON would be the way to go. Thanks again - hope all is well. Chris S
[0] https://leanpub.com/dataflowbook
I wonder if these problems have been successfully addressed somewhere.
I've built a few data flow systems in the past (my dissertation work [2] was based on one), but I decided to tolerate rather than avoid glitches early on because I found that way to be more resilient. Flapjax [3] avoids glitching via a topological sort, which I rather see as an optimization (to reduce unnecessary work) rather than as a correctness issue.
[1] http://research.microsoft.com/apps/pubs/default.aspx?id=2265...
[2] http://research.microsoft.com/pubs/179366/mcdirmid06superglu...
[3] http://cs.brown.edu/~sk/Publications/Papers/Published/mgbcgb...
Computing the dependency graph dynamically obviously leads to a lot more flexibility. I'm not really sure what you mean by "reactively" here: if you mean does it reactively update computations as state (and even code, if you want to get Bret Victor about it) changes, then ya.
Data flow by itself isn't necessarily reactive; actually the way it was originally defined to work very lazy push things through. Continuous interpretations where changes were propagated non-lazily didn't come until much later.
[1] https://github.com/openstack/taskflow
[2] http://docs.openstack.org/developer/taskflow/
https://code.google.com/p/papy/
Will have to have a closer look at this one as well, though.
Thanks for giving PaPy a try. Although PaPy [https://github.com/mcieslik-mctp/papy] has never gained traction :(, I have been using it daily for over 5 years (no bugs in the scheduler so far). By now our PaPy based genomics pipelines have probably processed petabytes of data.
Pretty diagram also @ https://wiki.openstack.org/wiki/TaskFlow#Big_picture ;)
https://groups.google.com/forum/#!forum/flow-based-programmi...
http://dataflowbook.com/cms
Can highly recommend.
Thank you for the kind words... everyone.
When I started writing the book "Dataflow and Reactive Programming Systems" I thought I completely understood the various forms of dataflow. Yet in doing research for the book I found there are many more subtle variations than I ever knew existed.
The book went through many iterations before I felt I accurately described the similarities and differences between the many forms of dataflow. I'm happy to hear that I may have succeeded.
https://groups.google.com/forum/#!searchin/flow-based-progra...
I'm not really familiar with how flow-based programming differs from data flow programming, however.
[1] http://clojure.com/blog/2013/06/28/clojure-core-async-channe...
[2] https://www.youtube.com/watch?v=6mTbuzafcII