News for hackers. Stuff that matters.
Re:The limitations of gradient descent as a principle of bra... by simonster on Sunday June 10, 2018 @02:08PM (#17279345)
tl;dr ordinary gradient descent is sensitive to the parametrization of the problem but natural gradient is not. This is an important fact (and one that is fairly well-known within the ML community), but it is not totally clear to me why it should be particularly relevant to neuroscience.
Re:The limitations of gradient descent as a principle of bra... by bjourne on Sunday June 10, 2018 @08:34PM (#17280924)
Isn't it only the trajectory that is sensitive to the parametrization? Afaik, the location of the minimas do not depend on it.
Re:The limitations of gradient descent as a principle of bra... by simonster on Tuesday June 12, 2018 @12:25AM (#17290696)
Yep, the minima are in the same locations. However, if the problem has multiple minima, then the parametrization can affect which minimum gradient descent actually reaches.