The dirty secret in ML is that logistic regression, SVM, and random forests often work better than deep learning on real problems.

This is literally how you can distinguish actual machine learners from blog readers.

- Deep learn everything!

vs.

- I took at look at the variable distributions, went with a forest model after transforming some of the data.

> I took at look at the variable distributions, went with a forest model after transforming some of the data.

That's what a statistician does too... I wish statistician word is more in vogue than data science or machine learners. Statistic is the discipline of data.

Uhhh most deep nets for classification or even regression have a regression at the end. You can even use a svm at the end. The power of deep learning is in the feature learning.

Yes, but using an SVM or whatever at the end of a deep network only really works for the same domains that deep networks work well in already - typically audio/visual data and some language datasets.

Most "simple" datasets like insurance prediction from structured data has been solved already via traditional techniques. The difficulty in machine learning was always in the noisy and unstructured datasets like images.

I'm curious what background you have in ML. "The difficulty in machine learning was always in the noisy and unstructured datasets like images" is truly a baffling statement to make.

"solved"

I'm going to disagree with you here.

For image classification, CNN's are still the way to go. But creating your own architecture and training your own novel model aren't necessary for most problems anymore it seems because of transfer learning.

I threw all of the above plus CNNs at MNIST problem and boosted decision trees outperformed CNNs.

Granted if I tuned both perfectly, CNNs probably would have outperformed but with defaults and a small amount of parameter search, boosting worked best.

That's pretty interesting, why are boosted decision trees so effective? I've heard the same meme applied to kaggle competitions (everything is just a way to shove data into xgboost, etc)

The grandparent post contains a hint of something I've already heard in lectures and from ML practicioners: boosting and random forests are more resilient to improper tuning (to put it another way, they are more universal and work well "out of the box").

Which, BTW, makes them more appealing to me personally. In many real-life cases extra few percent of accuracy matters very little, but ability to just apply something to a problem without much fuss matters a lot.

Gradient boosting also often does wonders.

You are absolutely right about that!

Deep Learning is mostly just about automatically creating good features. As an example if you had the driving data and were too lazy to hand engineer features for what looked like loops looking for parking. You could use deep learning and then use logistics regression ontop of the features you learnt

And you can explain the output in simple terms.

Just so you know, there are really powerful "classical statistics" classifier models! Like ordered or multinomial logit/probit which you can incorporate latent classes into, etc. Reference book for those is here[1]

[1] http://pages.stern.nyu.edu/~wgreene/DiscreteChoice/Readings/...

very true