Re:Machine Learning: Curse of Dimensionality
by deepnet on Wednesday August 05, 2015 @04:56PM
(#10012809)
And yet Goodfellow et al's paper only uses 2 hidden layers and 1 softmax output layer (on MNIST) which is not so very deep but the effect is still present.
On p.6 Goodfellow et al. compare the neural nets to linear solutions by removing the non-linear layers, so that a single linear function is learned and find for Deep Linear Networks that :
"
All minima are global minima, and are linked to each other in a continuous manifold.
"
Which is pretty exciting because the received wisdom was local-minima are a really big problem.
(Qualitative Characterisation of Neural Net Optimisation ICML 2015) http://arxiv.org/abs/1412.6544v4
On p.6 Goodfellow et al. compare the neural nets to linear solutions by removing the non-linear layers, so that a single linear function is learned and find for Deep Linear Networks that :
" All minima are global minima, and are linked to each other in a continuous manifold. "
Which is pretty exciting because the received wisdom was local-minima are a really big problem.