Re:Faster R with FastR
by tfehring on Thursday October 11, 2018 @12:02PM
(#18194629)
How large are we talking? I haven't had any problems with dplyr performance as long as my data fits in main memory. (I have 16GB, so that means single digit GB data frames at most - I realize that doesn't qualify as "very large".) It does slow down considerably for larger data sets, but I assumed that that was because it was hitting the pagefile.
[ Parent
| on HN ]
I have had the same experience with dplyr.
In the event that the data doesn't fit into memory, it's better to preprocess w/ SQL at the data-store level. There hasn't been a case where I'd need to feed massive amounts of data into a ggplot2 visualization unaggregated.