Re:What it takes to build great machine learning products
by sireat on Tuesday April 17, 2012 @09:16AM
(#3852393)
Our Data Mining (different field but somewhat related) professor had this quote "All models are wrong, but some models are useful" on the first day of university.
You can build an extremely complicated model that is not useful, where a simpler one might suffice.
Re:What it takes to build great machine learning products
by tel on Tuesday April 17, 2012 @12:18PM
(#3853322)
I've heard that line referred to as Box's Razor. It's definitely the right heuristic, but it's interesting to see that even if your model is right, you're still in trouble if it's too complex. This is a sort of bias/variance tradeoff.
No doubt this is a difficult position to master and those who perform well are able to tackle lots of mathematical and computational challenges. They also are model builders who (have tendency to) relentlessly seek complex models in order to solve complex problems.
The other, rarer side is the learning theorist who may or may not understand the model building, algorithmic, and computational tools but understands well the theories which allow us to have reasonable expectations that the tools of the first group will work at all. These guys have a funny story in that they were the old statisticians who got a major egg-on-the-face after proclaiming that essentially all of ML was impossible. Turns out the first group managed to redefine the problem slightly and make major headway (and money).
---
The thing I want to bring to light however is that the second group knows the math that bounds the capacities of ML algorithms. This isn't easy. It's one thing to say you recognize that the curse of dimensionality exists, but it's another to have felt it's mathematical curves and to build an intuition for what forces are sufficient to cause disruption.
The more experience you have with the learning maths, the more likely you are, I feel, to apply very simple algorithms, to be scared of "little x's" (real data) enough to treat it with great care, and to attempt to explore the problem space with confidence for what steps will lead you to folly.
---
It's a fine line between the two, though. Stray too far to the first group and you'll spend a month building an algorithm that does a millionth of a percentage point better than Fisher's LDA. Spend too much time in the second camp and you'll confidently state that no algorithm exists that does better than a millionth of a percentage point over Fisher LDA... and then lose purely by never trying.