News for hackers. Stuff that matters.

arun_sriniv

Re:Dataframes – Julia, R, Python on Wednesday December 31, 2014 @04:32AM (#8818722)
No worries :-). And glad to hear you're working on it! Let me know if I can be of any help.
Re:Dataframes – Julia, R, Python on Wednesday December 31, 2014 @03:50AM (#8818646)
ajinkyakale, "harder to learn" doesn't expose the fact that data.table provides so many features that, for example, dplyr just doesn't. And in addition, it is fast and memory efficient.

Rolling joins for example are slightly harder to grasp concept because most of us don't know what a "rolling" join is (unless you work regularly with time series).

Aggregating while joining is hard to grasp not because the syntax is hard, but the concept is inherently new.. It allows us to perform operations in a more straightforward manner, which most embrace after investing some time to understand it.

Binary search based subset, e.g., DT[J(4:6)] is again another concept that's new. One could use base R syntax and use vector scans to subset. But when you learn the difference between vector scans and binary search, you obviously don't want to vector scan. Now we can say that learning the difference between "vector scan" and "binary search" is really hard, but that'd be missing the point.

DT[x %in% 4:6] now internally uses binary search by constructing an index automatically! So you can keep using base R syntax.

And dplyr doesn't have any of these features.

In short, a huge part of "bit more time to get used" is due to data.table introducing concepts that aren't available in other tools/packages for faster and more efficient data manipulation. And I say this as a data.table user turned developer.

"harder to read after writing it" is very very subjective. I don't know what to say to that.

Re:Dataframes – Julia, R, Python on Tuesday December 30, 2014 @07:27AM (#8814289)
Unfortunately the datasets in that benchmark less than 3MB each in size - it fits entirely in cache. It doesn't give a good indication of how well the function/implementation scales on bigger data sizes that really matter (in terms of computation time, memory, how cache efficient it is etc..). How much does one really care about 0.018 vs 0.023 seconds?
Re:Dataframes – Julia, R, Python on Tuesday December 30, 2014 @06:21AM (#8814147)
@ajinkyakale, thanks. What'd be also interesting is to benchmark memory usage in addition to runtime.
Re:Dataframes – Julia, R, Python on Tuesday December 30, 2014 @06:15AM (#8814136)
data.table's `DT[i, j, by]` is quite consistent actually and is comparable to SQL's - i = where, j = select | update and by = group by.

This form is always intact. For example:

  require(data.table)  
  DT = data.table(x=c(3:7), y=1:5, z=c(1,2,1,1,2))

  DT[x >= 5, mean(y), by=z]        ## calculates mean of y while grouped by z on 
                                   ## rows where x >= 5

  DT[x >= 5, y := cumsum(y), by=z] ## updates y in-place with it's cumulative sum 
                                   ## while grouped by z on rows where x >= 5
"Harder to read after you've written it" and "harder to learn" are all very subjective and pointless. One could make very similar observations about `dplyr`, but I'll refrain from it here.

I implore the readers to take a look at over 100+ reviews on crantastic: http://crantastic.org/packages/data-table from users of the package.

Keeping `i`, `j` and `by` operations together allows optimising for speed and more importantly memory usage (altogether under a consistent syntax) - which are two very important aspects especially working on really huge data sets (10-100GB in RAM or more).

Here's a detailed benchmark (only on grouping so far) on 10 million (in MB) to 2 billion rows (100GB): https://github.com/Rdatatable/data.table/wiki/Benchmarks-%3A...

Re:The R language, for programmers on Friday December 05, 2014 @06:31AM (#8703962)
Here's a benchmark Matt recently did comparing data.table, dplyr and pandas on 50GB and 100GB: https://github.com/Rdatatable/data.table/wiki/Benchmarks-%3A...
Re:The R language, for programmers on Friday December 05, 2014 @06:26AM (#8703946)
We're in the process of adding more detailed vignettes illustrating more clearly the philosophy behind data.table's `i, j, by`. Should make things lot easier for beginners - https://github.com/Rdatatable/data.table/issues/944

data.table is not just fast, but is also more memory efficient - we want to highlight in the vignettes as well.

And timings are quite relevant even on 10 million rows: https://gist.github.com/arunsrinivasan/db6e1ce05227f120a2c9

For much larger data, check the project page: https://github.com/Rdatatable/data.table/wiki/Benchmarks-%3A...