News for hackers. Stuff that matters.
Re:Exact mapping between Variational Renormalization Group a... by charleshmartin on Tuesday November 29, 2016 @11:15PM (#13068449)
See my blog: https://charlesmartin14.wordpress.com/2015/04/01/why-deep-le...

I try to highlight the paper and some of the history and relevance, in a general way, but also hitting the math hard.

the more general idea, which I am still formulating, is that DL systems are very different from traditional ML (ala VC theory)

In traditional ML systems, we tune the regularizer to optimize the capacity of the learner

In deep learning, we would optimize both the capacity (entropy) of the learner, and the optimization problem (energy function)

This is also what happens in the stat mech of protein folding, where the energy is optimized, even when we are at minimum capacity. This gives rise to a funneled energy landscape

https://charlesmartin14.wordpress.com/2015/03/25/why-does-de...

Similar behavior is seen generally in stat mech near a critical point, and this is why the RG analogy is relevant for me.

We should be able to see this behavior if we simply plot the entropy vs energy of say an RBM. Im not entirely sure yet how general this is, but it works for MNIST.

I discuss (some of) this in more detail in a video from a talk I gave this summer at MMDS https://www.youtube.com/watch?v=kIbKHIPbxiU

Re:Exact mapping between Variational Renormalization Group a... by kdelok on Wednesday November 30, 2016 @09:28AM (#13070955)
You might be interested to speak with David Wales (http://www.ch.cam.ac.uk/person/dw34). He has recently been looking into properties of the landscapes of neural networks and has considerable experience in energy landscapes.
Re:Exact mapping between Variational Renormalization Group a... by dldl on Tuesday November 29, 2016 @11:48PM (#13068557)
Is it related to the predatory/prey population problems we did in Linear Algebra?

Something like this: http://www.math.umd.edu/~jmr/246/predprey.html

Re:Exact mapping between Variational Renormalization Group a... by mlechha on Wednesday November 30, 2016 @05:00AM (#13069560)
Interesting. I'm curious as to why you make that connection? What reminded you of predator prey systems?