Re:Apache Arrow 3.0
by CameronNemo on Wednesday February 03, 2021 @11:10PM
(#26022478)
Parquet is not Arrow. Parquet has optimizations for storage size at the expense of compute readiness. Arrow maximizes computation efficiency, at the expense of storage size.
Read more here: https://stackoverflow.com/questions/56472727/difference-betw...
I used arrow to read and write parquets across environments. Thats my dumb reason for finding arrow so useful.
Did you have to reformat the data to get that size saving?
Not at all. It was just swapping out readcsv with readparquet in R and Python. It was painless. Granted my dataset was mostly categorical so that’s why it compressed down so much, but it was a real lifesaver.
It was also nice to be able to read while bundles of parquet a into a single dataframe easily. So is nice for “sharding” really big parquets over multiple files. Or fitting under file size limits on git repos.
Re:Apache Arrow 3.0
by alisonkisk on Wednesday February 03, 2021 @11:35PM
(#26022644)
gzip probably would have been fine for that case even without parquet?
Gzip did help, but not as much as parquet. I don’t remember the exact sizes but I think parquet was was 3GB->20MB and gzip was like 3GB->180MB or something.
Also gzip was an extra step of unzip then read.
So it’s a good transport tool. It also happens to be fast to load and query, but I only used it because of the compact way it stores data without any hoops to jump through.
Of course one might say that it’s stupid to try to pass around a 2GB or 20MB file, but in my case I needed to do that.