Re:Exact mapping between Variational Renormalization Group a...
by mathgenius on Wednesday November 30, 2016 @03:01AM
(#13069178)
The RG used in statistical physics also suffers from being ad-hoc. There is a constant stream of physics papers that propose a new "architecture" for doing RG and comparing the numerics to other architectures, with very little (it seems) understanding of why these things do or do not work as well. For example: https://arxiv.org/abs/1412.0732
This paper suggests that there is a close relationship ("one-to-one mapping") between neural networks and something in physics called variational renormalization groups. They say:
> RG plays a central role in our modern understanding of statistical physics and quantum field theory. A central finding of RG is that the long distance physics of many disparate physical systems are dominated by the same long distance fixed points. This gives rise to the idea of universality – many microscopically dissimilar systems exhibit macroscopically similar properties at long distances Physicists have developed elaborate technical machinery for exploiting fixed points and universality to identify the salient long distance features of physics systems. It will be interesting to see, what, if any of this more complex machinery can be imported to deep learning.
If we can grasp some theory underpinning neural networks, the science of machine learning will be less "let's try this topography/architecture and see what happens" and will be more of "if we add a layer of this size in this part of the network, we will expect these results, or results to change in <specific way>".
Anyway, maybe physics can teach us something about neural networks.