An overview of gradient descent optimization algorithms

Re:An overview of gradient descent optimization algorithms by highd on Wednesday November 09, 2016 @04:31PM (#12914867)

This is primarily in regard to deep neural nets, where second-order methods are too expensive (O(p^2) in parameters, where p is on the order of millions).

I'm not sure conjugate gradient methods are used much, due to the non-convexity of the merit functions, and hard constraints aren't used much either.

[ Parent | on HN ]

Re:An overview of gradient descent optimization algorithms by partykid92 on Wednesday November 09, 2016 @04:43PM (#12914968)

quasi newton methods are not square in the dimension of the problem (think limited memory L-BFGS), and can be run in linear time. In my experience, however, they're 2-3 times slower than regular methods like ADAM.

[ Parent | on HN ]