Re:Apache Arrow 3.0
by BadInformatics on Wednesday February 03, 2021 @05:45PM
(#26020223)
Is that a problem in the Parquet format or in PyArrow? My understanding is that Parquet is primarily meant for on-disk storage (hence the default on-disk compression), so you'd read into Arrow for in-memory compression or IPC.
[ Parent
| on HN ]
I don't know whether it is a limitation of the format or of the implementation. I used the C++ library (or maybe it was C, I can't remember), which I assume PyArrow uses too.
> hence the default on-disk compression
No, Parquet doesn't support some compression formats that Arrow does.