News for hackers. Stuff that matters.
Re:Apache Arrow 1.0 by MrPowers on Monday July 27, 2020 @10:56PM (#23971862)
There is a project called vega that's a Spark implementation in Rust: https://github.com/rajasekarv/vega. It looks promising.

Databricks rewrote the Spark engine in C++ (called the Delta Engine, see here: https://databricks.com/blog/2020/06/24/introducing-delta-eng...).

Hopefully there will be some good alternatives that don't consume so much memory soon ;)

Re:Apache Arrow 1.0 by andygrove on Tuesday July 28, 2020 @10:01AM (#23975863)
There is also https://github.com/ballista-compute/ballista (shameless plug for my own project)
Re:Apache Arrow 1.0 by pjmlp on Tuesday July 28, 2020 @02:23AM (#23972867)
Hopefully they have all measures in place to prevent data to be tainted by C and C++ "features".