News for hackers. Stuff that matters.
Re:Apache Arrow 1.0 by choppaface on Monday July 27, 2020 @08:25PM (#23970945)
If you use pyspark and Spark for Parquet, you get:

* Easy type inference, even for nested maps and structs

* lz4 compression support

* SQL and directory partitioning out of the box

If you use pyarrow, you get: * Write support for nested types, but read support is broken / incomplete (it throws a TODO error)

* no lz4 support and a load of Jira politics blocking it

* You can query using Pandas (not SQL), but that querying can be much slower versus Spark (Spark is naturally parallelized)

pyarrow might need to hit 2.0.0 to be really viable. It’s definitely easier to use than parquet-mr though.

Re:Apache Arrow 1.0 by wesm on Tuesday July 28, 2020 @08:43AM (#23975089)
There is no “JIRA politics” blocking the LZ4 work, only a lack of volunteers to do the development and testing.
Re:Apache Arrow 1.0 by cheez on Monday July 27, 2020 @10:37PM (#23971760)
Hmm interesting. I will be investigating the performance of this bad boy.