Ballista: Distributed Compute with Rust, Apache Arrow, and Kubernetes

Re:Ballista: Distributed Compute with Rust, Apache Arrow, an... by diroussel on Wednesday July 17, 2019 @03:11PM (#20462972)

Andy, I see Spark as essentially a monad that lifts functions into it’s execution context, then sechedules the execution on remote nodes, and gathers the results.

How can you do that in an AOT language? Do you just express your functions at a higher level. Above the layer of containers?

[ Parent | on HN ]

Re:Ballista: Distributed Compute with Rust, Apache Arrow, an... by blaenk on Saturday July 20, 2019 @02:34AM (#20484466)

Consider Just-in-Time compilation which is very much used in AOT languages [0]. That could be one avenue, as long as the compiled code works on each machine (common instruction set), or maybe something slightly higher-level like LLVM IR can be produced and transmitted and then compiled on the driver/executor.

Another approach could simply be to transmit the function source to the driver, compile it, then distribute it to the executors assuming they're the same architecture, or compiling it on executors if not.

There are certainly options.

[0]: For example see https://llvm.org/docs/tutorial/BuildingAJIT3.html

[ Parent | on HN ]