Re:Feather: A Fast On-Disk Format for Data Frames for R and ...
by hadley on Tuesday March 29, 2016 @07:25PM
(#11385794)
It's often much faster than rds. And in the long long term there will be tools for computing on feather files that don't require loading it into memory. (In the short term I'll add ways to pull in slices of the full dataset)
Re:Feather: A Fast On-Disk Format for Data Frames for R and ...
by alsocasey on Tuesday March 29, 2016 @07:46PM
(#11385894)
Faster because it isn't (currently) using compression (which rds uses by default) or faster period?
Either way, the idea of mixed Python/R pipelines with feather file intermediates input/outputs is pretty sweet. Learn in scikit, save to feather, plot in ggplot2... using Make to tie the pieces together?
Re:Feather: A Fast On-Disk Format for Data Frames for R and ...
by hadley on Tuesday March 29, 2016 @07:54PM
(#11385930)
It's usually faster than either compressed or uncompressed RDS - but if you have heavily duplicated data, compressed RDS can be faster than feather (depending on some tradeoff between compression speed and disk speed). Feather will probably gain compression support eventually.
Is this like BigMemory but for data frames?
https://cran.r-project.org/web/packages/bigmemory/index.html
Thanks.