Ballista: Distributed Compute with Rust, Apache Arrow, and Kubernetes

Re:Ballista: Distributed Compute with Rust, Apache Arrow, an... by ohnoesjmr on Wednesday July 17, 2019 @04:00AM (#20457437)

It's total cost of ownership. Spark is a beast, half the resources is taken up by the platform itself.

[ Parent | on HN ]

Re:Ballista: Distributed Compute with Rust, Apache Arrow, an... by matteuan on Wednesday July 17, 2019 @04:16AM (#20457516)

Really? Can you explain better "half the resources is taken up by the platform itself". Do you have some numbers, experiments?

[ Parent | on HN ]

Re:Ballista: Distributed Compute with Rust, Apache Arrow, an... by ohnoesjmr on Wednesday July 17, 2019 @02:42PM (#20462729)

Running hadoop nodes, name nodes, yarn masters. These are massive complicated processes, that cumulatively are usually more complicated and expensive than the spark jobs themselves. At least from the cases I've seen.

[ Parent | on HN ]

Re:Ballista: Distributed Compute with Rust, Apache Arrow, an... by liancheng on Wednesday July 17, 2019 @10:03PM (#20466034)

Actually, none of these processes is essential for running Spark unless you have to read data from HDFS and have to run Spark on YARN.

Spark supports reading data from different sources, e.g., cloud blob storage, relational databases, NoSQL systems like C* and HBase. NameNode is only required if your data is stored on HDFS, and that is not an essential problem of Spark.

As for scheduling, Spark can run in standalone mode without any YARN components. Actually, that is how Spark clusters run in Databricks.

[ Parent | on HN ]