Re:Ballista: Distributed Compute with Rust, Apache Arrow, an...
by andygrove on Wednesday July 17, 2019 @01:50PM
(#20462166)
That's kinda what I'm trying to do here. Ballista is definitely very much inspired by Apache Spark but not a direct port. Spark has some design choices that were very much driven from being a JVM-first platform e.g. the way lambdas are serialized. When I eventually get to support custom user code in distributed execution I don't want it to be limited to Rust, but that's a topic for another blog post later this year.
[ Parent
| on HN ]
Re:Ballista: Distributed Compute with Rust, Apache Arrow, an...
by diroussel on Wednesday July 17, 2019 @03:11PM
(#20462972)
Andy, I see Spark as essentially a monad that lifts functions into it’s execution context, then sechedules the execution on remote nodes, and gathers the results.
How can you do that in an AOT language? Do you just express your functions at a higher level. Above the layer of containers?
Re:Ballista: Distributed Compute with Rust, Apache Arrow, an...
by blaenk on Saturday July 20, 2019 @02:34AM
(#20484466)
Consider Just-in-Time compilation which is very much used in AOT languages [0]. That could be one avenue, as long as the compiled code works on each machine (common instruction set), or maybe something slightly higher-level like LLVM IR can be produced and transmitted and then compiled on the driver/executor.
Another approach could simply be to transmit the function source to the driver, compile it, then distribute it to the executors assuming they're the same architecture, or compiling it on executors if not.
There are certainly options.
[0]: For example see https://llvm.org/docs/tutorial/BuildingAJIT3.html