Re:Why Deep Learning Works II: the Renormalization Group
by Animats on Sunday July 05, 2015 @02:31PM
(#9834423)
I got that feeling too, but I don't understand the subject well enough.
One key concept here is that you want your learning system to have some "free energy" metric which decreases as you run the training set through again and again. This insures some kind of convergence, rather than just thrashing around. Of course, the other problem is getting stuck at a local minimum, which is why you don't want to converge too fast. (I'm not up to speed on that; when I studied AI years ago, everybody was getting stuck at local minima. Now, that's less of a problem, and some of the old algorithms, run with slow learning rates over and over, get stuck less.)
The connection to physics seems to be from conservation of energy. The total energy of a system must decrease. For complex systems where energy is sloshing around from one form to another (flow, turbulence, gas compression, heat, maybe combustion) an energy-based analysis is a way of looking at the problem in a simpler way. That is apparently a useful way to look at deep learning systems. I think this is where the author is coming from.
Whether the connection to physical systems is useful, or merely an interesting analogy, isn't clear from the paper. It may be too soon to tell.
One key concept here is that you want your learning system to have some "free energy" metric which decreases as you run the training set through again and again. This insures some kind of convergence, rather than just thrashing around. Of course, the other problem is getting stuck at a local minimum, which is why you don't want to converge too fast. (I'm not up to speed on that; when I studied AI years ago, everybody was getting stuck at local minima. Now, that's less of a problem, and some of the old algorithms, run with slow learning rates over and over, get stuck less.)
The connection to physics seems to be from conservation of energy. The total energy of a system must decrease. For complex systems where energy is sloshing around from one form to another (flow, turbulence, gas compression, heat, maybe combustion) an energy-based analysis is a way of looking at the problem in a simpler way. That is apparently a useful way to look at deep learning systems. I think this is where the author is coming from.
Whether the connection to physical systems is useful, or merely an interesting analogy, isn't clear from the paper. It may be too soon to tell.