News for hackers. Stuff that matters.
Re:Machine Learning for Everyday Tasks by minimaxir on Tuesday November 22, 2016 @06:03PM (#13018462)
This is a case where knowing the data and it's inherent structure is just as important, if not more important, then knowing how to apply machine learning.

In this case, the formula was (time ~ content length + number of tags). Except, by construction, the content length is positively correlated with the number of tags! And since the parsing tags is the primary purpose of HTML scrapers, it's likely that content length is completely redundant or a poor predictor of time. (which was the conclusion). If you wanted to calculate the maginitude/significance of an explanatory variable, a linear regression is more than sufficient, and R has great tools for that.

The quantile plot revealed there is a skew in time processed, presumably the number of tags has a similar skew.

Re:Machine Learning for Everyday Tasks by thanatropism on Tuesday November 22, 2016 @06:16PM (#13018530)
I keep telling people in my profession who didn't really learn important things in school: "regression" should really be called "iid regression".

For example: time-series econometrics is a whole field dedicated to try to squeeze iid information from stuff that's not iid. So you can use regression.

But they're receptive to my message because they were repetitively warned that curve fitting was not regression analysis. Somehow this is lost in our "data" culture.

Re:Machine Learning for Everyday Tasks by thijsvandien on Tuesday November 22, 2016 @06:28PM (#13018604)
i.i.d.: independent and identically distributed
Re:Machine Learning for Everyday Tasks by ekianjo on Wednesday November 23, 2016 @12:56AM (#13020427)
> to try to squeeze iid information from stuff that's not iid. So you can use regression.

I'm not sure I understand your last sentence here - so you can, or you can't use regression for non iid stuff? It would seem that you cannot.

Re:Machine Learning for Everyday Tasks by pyromine on Wednesday November 23, 2016 @02:03AM (#13020701)
There is more than certainly forms of regressions that can work on heteroskedastic errors, i.e. non-iid (can be clustering, autoregressive, or more), but part of the general ordinary least squares assumptions and interpretations is iid. errors.

If you do not have iid. errors, any interpretations of the model will be skewed.

There are however other heteroskedasticaly robust interpretations of regressions.

Re:Machine Learning for Everyday Tasks by felixpl on Thursday November 24, 2016 @01:20AM (#13028583)
Weighted least squares should do the trick, i.e. scale your data with the inverse covariance matrix