News for hackers. Stuff that matters.

A comment on Deep Learning - The Biggest Data Science Breakthrough of the Decade. Parent comment

Re:Deep Learning - The Biggest Data Science Breakthrough of ... by jph00 on Thursday March 14, 2013 @04:45PM (#5377581)
DNNs can be thought of a stacked Restricted Boltzmann Machines. Their structure and training is very different to traditional MLPs. They derive in some ways from convolutional neural nets.

I describe some of the key differences between DNNs and MLPs in the webinar. Also, the webinar explains how recent advances go far beyond just applications to speech recognition - in particular I focus on a case study in chemoinformatics.

[ Parent | on HN ]
Re:Deep Learning - The Biggest Data Science Breakthrough of ... by ergodic on Thursday March 14, 2013 @08:20PM (#5378492)
>DNNs can be thought of a stacked Restricted Boltzmann Machines

Agree, as explained in Hinton et al 2006.

http://www.cs.toronto.edu/~hinton/absps/ncfast.pdf

But this is just for pre-training, as I said. If you look at Seides paper, they pre-train treating the MLP as a DBN and then they train it as a classic MLP with BP. Also using layer-wise BP pre-training does bring performance close to DBN pre-training, with no use of DBNs paradigms at all.

>Their structure and training is very different to traditional MLPs

I insist if we are talking of the same DNNs explained in Microsofts paper, this is not true. If we were to be talking about different DNNs please elaborate I would love to hear about that (seriously, no irony here).

[ Parent | on HN ]
Re:Deep Learning - The Biggest Data Science Breakthrough of ... by jph00 on Thursday March 14, 2013 @11:02PM (#5379018)
There's also the random knockout of neurons, as mentioned in the webinar.
[ Parent | on HN ]
Re:Deep Learning - The Biggest Data Science Breakthrough of ... by ergodic on Friday March 15, 2013 @05:54AM (#5379824)
I did not find that on the paper, are you referring to randomly switching off neurons?. I would be surprised if this would not be a technique of the original neural networks wave.
[ Parent | on HN ]