News for hackers. Stuff that matters.
Re:Apache Arrow 3.0 by didip on Wednesday February 03, 2021 @05:05PM (#26019783)
Question, doesn't Parquet already do that?
Re:Apache Arrow 3.0 by ianmcook on Wednesday February 03, 2021 @05:23PM (#26019974)
From https://arrow.apache.org/faq/: "Parquet files cannot be directly operated on but must be decoded in large chunks... Arrow is an in-memory format meant for direct and efficient use for computational purposes. Arrow data is... laid out in natural format for the CPU, so that data can be accessed at arbitrary places at full speed."
Re:Apache Arrow 3.0 by snicker7 on Wednesday February 03, 2021 @05:25PM (#26020000)
Yes. But parquet is now based on Apache Arrow.
Re:Apache Arrow 3.0 by ianmcook on Wednesday February 03, 2021 @05:44PM (#26020199)
Parquet is not based on Arrow. The Parquet libraries are built into Arrow, but the two projects are separate and Arrow is not a dependency of Parquet.
Re:Apache Arrow 3.0 by mr_toad on Wednesday February 03, 2021 @07:22PM (#26021072)
Arrow has definitely influenced the design of Parquet, they’re meant to compliment each other.