Re:How deep learning on GPUs wins datamining contest without...
by dave_sullivan on Friday November 02, 2012 @03:09PM
(#4734238)
The problem with using intuition for feature engineering is that intuition is often wrong. But more than that, it's much less scalable than a system that learns progressively higher level features from lower level data automatically.
And parameter selection isn't nearly as involved in practice--but it is important to know what you're doing because you'll basically be translating from papers to code in order to get a working implementation, at least until some startup comes out with a plug and play deep learning paas
For anyone interested in playing with the tools that are available, deeplearning.net provides a set of very good tutorials with working code.
Re:How deep learning on GPUs wins datamining contest without...
by doobwa on Friday November 02, 2012 @02:42PM
(#4734116)
I agree these methods still require a fair amount of expert knowledge and intuition in order to make the various choices you mention. On the other hand, Bayesian optimization can prove useful for exploring such a space. A recent paper (http://arxiv.org/pdf/1206.2944.pdf) used Bayesian optimization with GPs to find hyperparameter settings for a deep convolutional network. The resulting hyperparameters gave state of the art performance, beating those chosen by an expert (Krizhevsky, the researcher who recently won ImageNet).
Re:How deep learning on GPUs wins datamining contest without...
by danger on Friday November 02, 2012 @02:51PM
(#4734161)
Agreed. Here's a previous HN discussion on that topic:
http://news.ycombinator.com/item?id=4281630
Re:How deep learning on GPUs wins datamining contest without...
by marshallp on Friday November 02, 2012 @03:01PM
(#4734213)
Is there a study comprehensively comparing the results of those different settings on a variety of data sets. (your opinion is commonly stated but I can't find any thorough evidence to it)
A common criticism with these methods is that they merely shift engineering from features to parameters that specify the architecture. There are many choices to be made: The exact number of layers, number of neurons per layer, the connectivity, sparsity parameters, non-linearities, sizes of receptive fields, learning rates, weight decays, pre-training schedule etc etc etc. Perhaps even worse, while you can use intuition to design features, it is not as trivial to see if you should be using a sigmoid, tanh, or rectified linear units (+associated parameters for each) in the 3rd layer of the network. And maybe even worse, these parameters can actually have quite a strong effect on the final performance.
These are still powerful models and we are learning a lot about what works and what doesn't (and I'm optimistic) but don't make the mistake of thinking they are automatic. For now, you need to know what you're doing.