Re:Using Python generators for real work [pdf]
by davidhollander on Sunday January 23, 2011 @07:06PM
(#2133734)
Here is some older material on pipeline coding in Lua you might be interested in that describes this problem in terms of Filters, Sources, and Sinks: http://lua-users.org/wiki/FiltersSourcesAndSinks
LuaSocket implementation of the above: http://w3.impa.br/~diego/software/luasocket/ltn12.html
It would be interesting to combine pipelines into something that branches off into various categories. For e.g., you could split the lines of access-log file into IP addresses and size of requests and fork off separate processing threads: one for obtaining unique set of IP addresses and another for summing up.
It seems like all that is needed for describing a pipeline is:
At the face of it, it looks similar to Apple's Automator and Matt Welsh's PhD thesis on SEDA: EDIT: Formatting