News for hackers. Stuff that matters.
Re:Top ten algorithms in data mining (2007) [pdf] by mooneater on Tuesday December 18, 2012 @02:58PM (#4939267)
Missing Logistic Regression: so old school (from the 1930s), so mathematically rigorous (compare to neural networks for example), so powerful, and so often overlooked.
Re:Top ten algorithms in data mining (2007) [pdf] by dxbydt on Tuesday December 18, 2012 @06:11PM (#4940187)
In the textbook "multivariate stats" by izenman, (http://www.amazon.com/gp/product/0387781889/ ) , he claims that stats & ML progressed in parallel. So traditional stats techniques like OLS, multiple regression, nonlinear regression, logistic regression, GLMs are generally not covered in ML. Similarly ML topics like k-means, svm, random forests etc. are not taught by the stats dept.

What is happening in this past decade is a convergence of stats & ML, primarily driven by data scientists working in the domain of big data. The stats folks are slowly incorporating ML techniques into stats & finding rigorous heuristics for when they should be employed. Similarly ML guys, who are mostly CS folk who unfortunately have taken only 1 course on undergraduate stats & probability, are discovering you can do so much more without resorting to needless large-scale computation, by sampling intelligently & leveraging plain old statistics.

This schism between stats & ML can be leveraged very profitably during interviews :))

When I interview data science folks, I usually ask very simple questions from plain stats - how would you tell if a distribution is skewed...if you have an rv X with mean mu, and say rv Y = X-mu, then what is the mean of Y...if you have an rv A with mean 0 variance 1, then what are the chances of being 3 standard deviations away from the mean if you have no clue about the distribution of A ? What if you knew A was unimodal ? What if A is now normally distributed ?

Now if its a stats guy, I ask very simple ML....what is perceptron, have you heard of an neural network etc.

surprisingly, the stats guys do much better on ML than the ML guys on stats!

Re:Top ten algorithms in data mining (2007) [pdf] by malkarouri on Tuesday December 18, 2012 @07:32PM (#4940558)
Not surprising, really. ML is the shiny new thing, so the MLers don't tend to feel they missed anything while the statisticians need to keep up with the times.

I say this as an MLer still struggling to find out what R^2 is, among other things ..

Re:Top ten algorithms in data mining (2007) [pdf] by dxbydt on Tuesday December 18, 2012 @08:13PM (#4940680)
Its just a stupid fraction. say you have a dataset ie. sequence of (x,y) tuples. In OLS, you try to fit a line onto the dataset. So your manager wants to know how well the line fit your dataset. If it does a bang-up job, you say 100% aka rsquare of 1. If it does a shoddy job, you say 0% aka rsquare of 0. Hopefully your rsq is much closer to the 1 than to the 0.

Here I just coded up a 10-liner for you: https://gist.github.com/4333595

Re:Top ten algorithms in data mining (2007) [pdf] by _dps on Wednesday December 19, 2012 @12:05AM (#4941237)
Respectfully, it's not a stupid fraction. It is a fundamental quantity arising from the linear algebraic interpretation of correlation.

Correlation induces an inner product on the set of zero-mean random variables. The regression coefficient is precisely the projection coefficient <x,y>/<y,y> and R^2 is precisely the Cauchy-Schwarz ratio <x,y>^2 / <x,x><y,y> (i.e. the product of the two projection coefficients between x and y).

It is a theoretically natural measure of linear quality-of-fit. It has the added bonus of being equal to the ratio of modeled variance to total variance (variance being the square-norm of a random variable in the norm induced by the correlation inner product).

It's also very very cheap to compute. Though there are more practically useful measures of "predictive power", like mutual information, R^2 does an admirable job for an O(1)-space and O(num data)-time predictiveness metric.

Re:Top ten algorithms in data mining (2007) [pdf] by politician on Wednesday December 19, 2012 @01:26AM (#4941440)
> Respectfully, ...

I suspect that I learned more about R^2 by reading these comments in the order presented (informal, then formal) than I would have had they been reversed.

Re:Top ten algorithms in data mining (2007) [pdf] by textminer on Tuesday December 18, 2012 @03:50PM (#4939508)
Learning about the basis of logistic regression in Ng's Stanford class was eye-opening. Also like that he then motivated generalized linear models, and why they're nice (e.g., parameters are linear with input data; the maximum-likelihood hypothesis is the expectation of the sufficient statistic) and he explains why we see the logistic function in so many damn places (it's the response function whenever y|x is distributed as an exponential family).
Re:Top ten algorithms in data mining (2007) [pdf] by alook on Tuesday December 18, 2012 @04:30PM (#4939696)
100% agree.

It was great how he spent a lot of time on logistic regression before delving into SVM's or Neural Nets - it was much easier to understand the cost functions & regularization for other types of classifiers after having understood those for logistic regression.

My takeaway: if you can avoid adding risk to your systems by using more complicated models, you should.

Re:Top ten algorithms in data mining (2007) [pdf] by yid on Tuesday December 18, 2012 @03:34PM (#4939446)
Neural networks are well understood as additive models. They may not have nice closed form training algorithms, but they are also more expressive than logistic regression.
Re:Top ten algorithms in data mining (2007) [pdf] by psb217 on Tuesday December 18, 2012 @11:01PM (#4941071)
Conceptually, logistic regression is a simple neural network. A neural network with no hidden layers and a softmax output layer using cross-entropy loss is precisely equivalent to logistic regression. Though, if you were to extend your suggestion of logistic regression to include its locally-weighted variants, I'd gladly support your cause.
Re:Top ten algorithms in data mining (2007) [pdf] by tocomment on Tuesday December 18, 2012 @03:13PM (#4939336)
Can you explain more about why you like it?
Re:Top ten algorithms in data mining (2007) [pdf] by fiter on Tuesday December 18, 2012 @08:16PM (#4940686)
In the paper they mention logistic regression as an extension of the naive Bayes model (see page 26).

"Another extension to the naive Bayes model was developed entirely independently of it. This is the logistic regression model."