Re:Geoffrey Hinton: Introduction to Deep Learning, Deep Beli...
by rudyl313 on Monday December 14, 2015 @05:57PM
(#10734642)
Unfortunately this talk is kind of dated already. Most people don't stack RBMs or autoencoders to pretrain the weights anymore. If you use dropout with rectified linear units, you don't have to pretrain, even for large architectures.
[ Parent
| on HN ]
Re:Geoffrey Hinton: Introduction to Deep Learning, Deep Beli...
by bradneuberg on Monday December 14, 2015 @06:56PM
(#10734911)
It's not just ReLUs that have helped, its also better random initialization before starting training, such as using Xavier initialization (http://andyljones.tumblr.com/post/110998971763/an-explanatio...)
Also, batch normalization helps with convergence as well. In addition, LSTMs work when dealing with recurrent neural nets.