Scientists See Promise in Deep-Learning Programs

Re:Scientists See Promise in Deep-Learning Programs by iskander on Saturday November 24, 2012 @02:15PM (#4826146)

Are you talking about Hinton's "A Fast Learning Algorithm for Deep Belief Nets"? Before that was published, Hinton's lab and their spiritual allies were training large restricted boltzmann machines via truncated sampling for decades. And Yann LeCun's convolutional networks (the architecture used in Google's vision project) have also been trained via plain old stochastic gradient descent for decades.

As far as I can tell there hasn't been any single revolutionary breakthrough in this field...we just keep getting more computing power, discovering better tricks and heuristics, and trying to build larger and larger networks.

[ Parent | on HN ]

Re:Scientists See Promise in Deep-Learning Programs by dimatura on Saturday November 24, 2012 @02:44PM (#4826232)

I'm guessing the "pretraining" described in this 2006 Science article: http://www.cs.toronto.edu/~hinton/science.pdf. (Possibly the same line of research the article you mention). Sure, if you look at things from a wide enough perspective, there haven't been any "revolutionary" breakthroughs. But this did seem to reignite interest in neural nets after they had sort of languished for a while. (Science described this work, somewhat hyperbolically, as "Neural nets 2.0").

[ Parent | on HN ]

Re:Scientists See Promise in Deep-Learning Programs by iskander on Saturday November 24, 2012 @03:12PM (#4826315)

I think culturally, Hinton made a big splash and got people to pay attention to learning hierarchies and SGD-like training algorithms. Algorithmically, though, SGD is both ancient and still the dominant deep learning training technique (though useful tricks, extensions, and rules of thumb keep accumulating)

[ Parent | on HN ]