News for hackers. Stuff that matters.

A comment on Apache Arrow 3.0. Parent comment

Re:Apache Arrow 3.0 by prepend on Wednesday February 03, 2021 @09:43PM (#26021972)
Not at all. It was just swapping out readcsv with readparquet in R and Python. It was painless. Granted my dataset was mostly categorical so that’s why it compressed down so much, but it was a real lifesaver.

It was also nice to be able to read while bundles of parquet a into a single dataframe easily. So is nice for “sharding” really big parquets over multiple files. Or fitting under file size limits on git repos.

[ Parent | on HN ]
Re:Apache Arrow 3.0 by alisonkisk on Wednesday February 03, 2021 @11:35PM (#26022644)
gzip probably would have been fine for that case even without parquet?
[ Parent | on HN ]
Re:Apache Arrow 3.0 by prepend on Thursday February 04, 2021 @07:03AM (#26024640)
Gzip did help, but not as much as parquet. I don’t remember the exact sizes but I think parquet was was 3GB->20MB and gzip was like 3GB->180MB or something.

Also gzip was an extra step of unzip then read.

[ Parent | on HN ]