News for hackers. Stuff that matters.
Re:Why Julia's DataFrames Are Still Slow by IndianAstronaut on Sunday November 29, 2015 @09:21AM (#10644613)
One thing I would really like to see happen is out of core data and statistics in Julia just like SAS. Not possible in either R or Python.
Re:Why Julia's DataFrames Are Still Slow by heydenberk on Sunday November 29, 2015 @09:54AM (#10644708)
If I understand you, this is indeed possible and has been recently implemented in Python. Take a look at dask: http://dask.pydata.org/en/latest/

> Users interact with dask either by making graphs directly or through the dask collections which provide larger-than-memory counterparts to existing popular libraries:

Re:Why Julia's DataFrames Are Still Slow by digitalzombie on Sunday November 29, 2015 @10:27AM (#10644800)
Just to clarify, OP statement of "out of the core" means concurrent processes?
Re:Why Julia's DataFrames Are Still Slow by chubot on Sunday November 29, 2015 @12:26PM (#10645275)
I think he means operating on data that doesn't fit in memory, and as mentioned I think R and Python both have packages for that.
Re:Why Julia's DataFrames Are Still Slow by IndianAstronaut on Sunday November 29, 2015 @11:19PM (#10647671)
Dask looks quite interesting. Though I am running into some issues with merges and such.

My temporary fix is to use the chunking functionality of Pandas.

Re:Why Julia's DataFrames Are Still Slow by johnmyleswhite on Sunday November 29, 2015 @10:35AM (#10644817)
There's work being done to wrap Dato's SFrames library, which would solve that problem.
Re:Why Julia's DataFrames Are Still Slow by tadlan on Sunday November 29, 2015 @10:37AM (#10644825)
That's just the data structure. Fitting models would be much tougher.
Re:Why Julia's DataFrames Are Still Slow by johnmyleswhite on Sunday November 29, 2015 @03:53PM (#10646021)
That really depends on how you fit models. Maximum likelihood estimation (or MAP estimation) for models of IID database rows is often trivial to implement using an iterative algorithm that applies to both in-memory and out-of-core data stores.
Re:Why Julia's DataFrames Are Still Slow by Nrpf on Sunday November 29, 2015 @11:04PM (#10647619)
You mean IRLS or SGD or something like that?
Re:Why Julia's DataFrames Are Still Slow by johnmyleswhite on Sunday November 29, 2015 @11:49PM (#10647777)
I would use SGD when I can tolerate an inexact estimator. For problems when I need an exact MLE (because, for example, I need to invoke asympotic normality to get CI's), I can usually do exact computations using less memory than IRLS would require.

In particular, what I had in mind is that most black-box optimization tools (e.g. L-BFGS) only need an implementation of a function and its gradient -- both of which can be accumulated iteratively using O(P) memory and O(N * P) time per iteration for any data store that can stream all N observations. Given code that can accumulate function values and gradients, an optimization method like L-BFGS can estimate parameters for many kinds of models. In my experience, there's often no need to materialize the kinds of dense matrices that are typically used in R's default approach to GLM model fitting.