News for hackers. Stuff that matters.
Re:The Big Data Brain Drain: Why Science is in Trouble by michaelt on Saturday November 09, 2013 @12:26PM (#6702283)
How would one take a mean of n elements without visiting all n elements? Won't the memory bandwidth and big-O complexity always be the same? Genuinely curious.
Re:The Big Data Brain Drain: Why Science is in Trouble by CamperBob2 on Saturday November 09, 2013 @02:09PM (#6702713)
The language used in MATLAB and Octave is designed for vector processing to an extent most developers haven't seen before. MATLAB doesn't mean "Math Laboratory", it means "Matrix Laboratory." Operations on row and column vectors are first-class language elements. You almost never have to manually iterate over an array to compute its statistics -- you'd just say M = mean(A [,dim]) where A is a standalone vector or a column vector of a matrix. In that example, M itself is a vector, if A was a matrix.

MATLAB syntax is ugly but the underlying principles are pretty cool. Well-written code scales automatically on newer hardware, or at least it has the potential to. That's not true in languages where higher-order vectors are built from discrete scalars.

Re:The Big Data Brain Drain: Why Science is in Trouble by joe_the_user on Saturday November 09, 2013 @09:58PM (#6704939)
The good stuff of Matlab must be balanced by it's perverse, pathological and obscene qualities.

The most vile aspect of Matlab is the faith every researcher has that producing something in Matlab is enough when the reality is code coming from Matlab will never escape, will never be as useful nakin-style pseudo for the creation of any larger system.

Re:The Big Data Brain Drain: Why Science is in Trouble by hharrison on Saturday November 09, 2013 @03:23PM (#6703025)
In MATLAB, R, or numpy, it's the difference between `mean(n)` and manually looping. It's not an issue of algorithmic efficiency, it's an issue of lost productivity because they don't even write a function to reuse (all they understand is scripting) they recode the loop every single time they have to sum or take the mean of something.
Re:The Big Data Brain Drain: Why Science is in Trouble by gaius on Sunday November 10, 2013 @03:28PM (#6708283)
Well it is because NumPy and friends do all the heavy-lifting in hand-tuned C. dis() your Python function for taking a mean and see the difference, it's huge.
Re:The Big Data Brain Drain: Why Science is in Trouble by ScottBurson on Saturday November 09, 2013 @01:32PM (#6702530)
The point is not computational time; the point is that one could simply call an existing library function rather than hand-coding the loop oneself and risking making an error (a fencepost error, for example).
Re:The Big Data Brain Drain: Why Science is in Trouble by ivoflipse on Saturday November 09, 2013 @02:14PM (#6702725)
I can understand that you'd want to manually check what's happening. For example taking the mean over the rows of a 2D array using numpy's mean function and aren't really sure whether axis=0 or axis=1 refers to the rows.

But you'd only have to figure it out once and then learn to trust numpy, instead of rolling your own version every time.

Re:The Big Data Brain Drain: Why Science is in Trouble by darkarmani on Saturday November 09, 2013 @01:58PM (#6702667)
You missed these key words: "manually iterating"

So looping in a high-level language rather than using vectorized functions.

Re:The Big Data Brain Drain: Why Science is in Trouble by aidos on Saturday November 09, 2013 @01:24PM (#6702497)
It's probably more in reference to the layer the work is completed in. I haven't used matlab in years but you can probably sum an array by iterating or you can call a faster more efficient library. You get much greater gains when doing this in higher dimensions. If you can do your operations at a matrix level you get a magnitude improvement in speed in most languages.
Re:The Big Data Brain Drain: Why Science is in Trouble by binarysolo on Saturday November 09, 2013 @01:05PM (#6702426)
I think the concern is over the manual component of it, especially if that set of n is big by human standards. (Say, doublechecking a few hundred entries of some column entry by calculator.)