News for hackers. Stuff that matters.
Re:Faster R with FastR by WhompingWindows on Thursday October 11, 2018 @10:48AM (#18193947)
"Moreover, support for dplyr and data.table are on the way. "

Well, I can't really use it in my day to day work, since that almost always involves cleaning and munging via one of those two packages. And it's not like ggplot2 is where my R code is most delayed, usually I'm working on aggregate data or perhaps a very much smaller analytical dataset which requires much less speed for plotting. My hang-ups are in initial munging phases where the data is still very large, which often calls for data.table over dplyr due to the latter's much slower performance.

Re:Faster R with FastR by ekianjo on Thursday October 11, 2018 @11:19AM (#18194210)
Yeah, data.table provides already significant speedup vs dplyr - so much that the "better" syntax of dplyr makes no sense anymore when you have to deal with very large datasets. But maybe FastR can somewhat change that?
Re:Faster R with FastR by baldfat on Thursday October 11, 2018 @01:43PM (#18195505)
Wait so the time difference of you running your code is longer then working with a "better syntax?"

I spend hours cleaning up data and only have to run the code once (I normally save the output to a feather and then work with a separate file from there).

I still believe that the 'tidyverse' is hands down the best thing that has happened to R and is the whole reason why R has grown so fast.

Re:Faster R with FastR by WhompingWindows on Thursday October 11, 2018 @02:00PM (#18195671)
Sometimes it can take 12 or more hours to run the code on the millions of observations. There's also competition from other researchers who use computational resources, which can mean I have to leave something running for hours due to the server being heavily queried. My workflow also doesn't allow easy interruption of the execution, sometimes it has to execute fully incorrectly before I can change an error or parameter.
Re:Faster R with FastR by baldfat on Friday October 12, 2018 @11:43AM (#18202201)
I would then say your using the wrong tool for your problem? I can't imagine 12 hour runs. I would imagine Spark is a better bet or is that not an option?

https://spark.apache.org/docs/latest/sparkr.html

Re:Faster R with FastR by semi-extrinsic on Thursday October 11, 2018 @04:38PM (#18196855)
I dunno how large your data set is, but I just set up for work a 16-core Threadripper workstation with 32 GB RAM and 1 TB M.2 SSD for approx. $2500. If it can regularly save you hours or days of waiting, getting something equivalent should be a no-brainer.
Re:Faster R with FastR by tfehring on Thursday October 11, 2018 @12:02PM (#18194629)
How large are we talking? I haven't had any problems with dplyr performance as long as my data fits in main memory. (I have 16GB, so that means single digit GB data frames at most - I realize that doesn't qualify as "very large".) It does slow down considerably for larger data sets, but I assumed that that was because it was hitting the pagefile.
Re:Faster R with FastR by minimaxir on Thursday October 11, 2018 @12:16PM (#18194755)
I have had the same experience with dplyr.

In the event that the data doesn't fit into memory, it's better to preprocess w/ SQL at the data-store level. There hasn't been a case where I'd need to feed massive amounts of data into a ggplot2 visualization unaggregated.

Re:Faster R with FastR by steve_s on Thursday October 11, 2018 @04:13PM (#18196694)
FastR doesn't alter the semantics of R, so when dplyr copies a vector in GNU-R then FastR has to copy it too. However, FastR does use reference counting (not sure if that's turned on in GNU-R 3.5.1 now) so it may avoid some unnecessary copies.
Re:Faster R with FastR by jsmith99 on Thursday October 11, 2018 @12:00PM (#18194603)
You can use dplyr syntax on data tables, usually with data table speed, especially if you load dtplyr.
Re:Faster R with FastR by claytonjy on Thursday October 11, 2018 @12:08PM (#18194696)
How is dtplyr on the memory aspect? Doesn't it force dplyr-style deep copies?
Re:Faster R with FastR by hadley on Friday October 12, 2018 @02:26PM (#18203563)
dplyr does not do deep copies. See discussion in https://adv-r.hadley.nz/names-values.html#df-modify
Re:Faster R with FastR by RosanaAnaDana on Thursday October 11, 2018 @11:24AM (#18194260)
Yeppo. Also, for myself at least in the geospatial realm, I need raster, rgdal, sp, sf, and parallel. The primary allure of R (imo) is the thousands of packages that allow you to quickly and easily implement whatever you want to do. Combine those with data.table and parLapply, and you're off to the races.