News for hackers. Stuff that matters.
Re:Why Deep Learning Works II: the Renormalization Group by charleshmartin on Sunday July 05, 2015 @02:21PM (#9834380)
'Technically' this is correct--the RBM CD algo is not minimizing this function; that's not the point.

It is known that when training an RBM, the reconstruction error decreases but not monotonically; in fact it fluctuates. In the words of Hinton, 'trust it but don't use it'.

http://www.cs.toronto.edu/~hinton/absps/guideTR.pdf (which is cited in the post as well)

So in a global sense, yes, I would say that the RBM does eventually minimize the reconstruction error even though it fluctuates.

I can even offer a conjecture here on why the error fluctuates ; in a discrete RG flow map, there could be finite size effects that would give log-periodic fluctuations. This is a stretch--but it is something that could be tested.

I explain this idea here http://charlesmartin14.wordpress.com/2015/01/16/the-bitcoin-...

As to stacking the RBMs to form a DBN--yeah that's the point. "Hinton showed that RBMs can be stacked and trained in a greedy manner to form so-called Deep Belief Networks (DBN)" http://deeplearning.net/tutorial/DBN.html

There are no comments on this comment yet.