News for hackers. Stuff that matters.

A comment on Performing Linear Regression Using Ruby. Parent comment

Re:Performing Linear Regression Using Ruby by textminer on Tuesday September 11, 2012 @10:57PM (#4508908)
Python is surprisingly heavy-duty. But my kingdom for a seamlessly distributed or parallelized version of NumPy/SciPy! How nice would it be to just enter "C = A * B", with A living as a sparse CSC across many nodes?
[ Parent | on HN ]
Re:Performing Linear Regression Using Ruby by msellout on Tuesday September 11, 2012 @11:10PM (#4508946)
Would Disco (http://discoproject.org/) work for you?
[ Parent | on HN ]
Re:Performing Linear Regression Using Ruby by cdavid on Wednesday September 12, 2012 @06:55AM (#4510060)
I don't think MR is a good abstraction for implementing linear algebra, and I expect the overhead to be too high (although I don't have numbers to back that up). For large problems (>> couple of machines worth of RAM), you use big iron HPC solutions, or you avoid 'exact' linear algebra altogether to focus on one-pass algorithms.

For example, instead of computing an exact SVD, you will use something like Hebbian algorithm to compute the SVD in a streaming manner (that's what Mahaout implements for example).

[ Parent | on HN ]
Re:Performing Linear Regression Using Ruby by Radim on Wednesday September 12, 2012 @04:59AM (#4509791)
No, the sparse matrix code in SciPy is plain C (not even multi-core, let alone distributed).

EDIT: or did you mean Disco offers distributed sparse CSC operations?

[ Parent | on HN ]
Re:Performing Linear Regression Using Ruby by hogu on Tuesday September 11, 2012 @11:41PM (#4509022)
we, http://continuum.io/, are working on this.
[ Parent | on HN ]