News for hackers. Stuff that matters.
Re:Apache Arrow 1.0 by mumblemumble on Monday July 27, 2020 @10:09PM (#23971627)
I can't speak to the relative merits of Julia, but I am honestly interested in anything that seeks to produce a less memory-hungry alternative to Spark.

Rust, to me, seems like a natural enough choice. It is easy to mate it to other languages, including all the major data science ones, so it would theoretically work well as the basis for a distributed compute engine that has good support for all of them as client languages. Would the same work for Julia? IIRC, it's a bytecode compiled language, which I imagine would make it difficult to link Julia libraries from other technology stacks.

Re:Apache Arrow 1.0 by MrPowers on Monday July 27, 2020 @10:56PM (#23971862)
There is a project called vega that's a Spark implementation in Rust: https://github.com/rajasekarv/vega. It looks promising.

Databricks rewrote the Spark engine in C++ (called the Delta Engine, see here: https://databricks.com/blog/2020/06/24/introducing-delta-eng...).

Hopefully there will be some good alternatives that don't consume so much memory soon ;)

Re:Apache Arrow 1.0 by andygrove on Tuesday July 28, 2020 @10:01AM (#23975863)
There is also https://github.com/ballista-compute/ballista (shameless plug for my own project)
Re:Apache Arrow 1.0 by pjmlp on Tuesday July 28, 2020 @02:23AM (#23972867)
Hopefully they have all measures in place to prevent data to be tainted by C and C++ "features".
Re:Apache Arrow 1.0 by pjmlp on Tuesday July 28, 2020 @02:21AM (#23972849)
It is the bytecode compiled language like any other that uses LLVM as their backend.

The alternative to Spark will be Spark, as JVM slowly improves their memory APIs from release to release.

People tend to hand wave how much it costs to reboot an ecosystem.