Apache Arrow 3.0

Re:Apache Arrow 3.0 by IshKebab on Wednesday February 03, 2021 @05:26PM (#26020022)

Yeah I do. For example Apache Arrow supports in memory compression. But Parquet does not support that. I had to look through the code to find that out, and I found many instances of basically `throw "Not supported"`. And yeah as I said the documentation is just non-existent.

If you are already using Arrow, or you absolutely must use a columnar file format then it's probably a good option.

[ Parent | on HN ]

Re:Apache Arrow 3.0 by BadInformatics on Wednesday February 03, 2021 @05:45PM (#26020223)

Is that a problem in the Parquet format or in PyArrow? My understanding is that Parquet is primarily meant for on-disk storage (hence the default on-disk compression), so you'd read into Arrow for in-memory compression or IPC.

[ Parent | on HN ]

Re:Apache Arrow 3.0 by IshKebab on Thursday February 04, 2021 @01:36PM (#26028357)

I don't know whether it is a limitation of the format or of the implementation. I used the C++ library (or maybe it was C, I can't remember), which I assume PyArrow uses too.

> hence the default on-disk compression

No, Parquet doesn't support some compression formats that Arrow does.

[ Parent | on HN ]