Re:Graph: Faster Abstractions for Structured Computation
by sigil on Wednesday May 01, 2013 @03:15PM
(#5640164)
How does the parallel "execution strategy" work? Do you specify a maximum concurrency level? Can you distribute the computation onto other machines?
[ Parent
| on HN ]
Re:Graph: Faster Abstractions for Structured Computation
by w01fe on Wednesday May 01, 2013 @03:53PM
(#5640365)
Graph is currently just in-process (no cross-machine distribution), although it's definitely a possibility down the line. The currently released parallel strategy is also just a pedagogical example, not really meant to be used.
Here's some WIP on a more practical parallel compilation: https://gist.github.com/w01fe/4710008 Once the kinks get worked out this will go into the OSS project. Presumably concurrency level will be controlled by a parameter, and/or passing an appropriate ExecutorService.
Re:Graph: Faster Abstractions for Structured Computation
by jared314 on Wednesday May 01, 2013 @09:05PM
(#5641882)
It would be very interesting if you could easily convert a graph into a Storm[0] topology. An example of the Storm Clojure DSL is available in the storm-starter project[1].
[1] https://github.com/nathanmarz/storm-starter/blob/master/src/...
Re:Graph: Faster Abstractions for Structured Computation
by batbomb on Wednesday May 01, 2013 @03:58PM
(#5640396)
I'm not sure how it's implemented in Graph, but the best thing you can do is to abstract out an execution to a general object.
From there, you can create all sorts of adapters and the "graph" engine just farms out the execution to the appropriate adapter. This could be anything from a job daemon on a batch farm, or, say, a typed ThreadPoolExecutor on a machine (which can either execute native clojure/java, or any sort of other script language.
Re:Graph: Faster Abstractions for Structured Computation
by thelittlelisper on Wednesday May 01, 2013 @06:32PM
(#5641267)
This would be brilliant. Does any other framework support this? (outside Clojure)