Side note, anybody aware of any implementations of the "learning to learn gradient descent by gradient descent" work [0]? I'd love to boost my training time, but I'm worried implementing it myself will just result in an additional set of hyperparameters to tune.
Re:An overview of gradient descent optimization algorithms
by zo7 on Wednesday November 09, 2016 @04:17PM
(#12914753)
Is it possible to swap the link on this post? That's kind of silly that they just copy the original post. Understanding different optimizers and why they work is important though, especially since often people will just use them without considering what they're actually doing.
Re:An overview of gradient descent optimization algorithms
by highd on Wednesday November 09, 2016 @04:31PM
(#12914867)
This is primarily in regard to deep neural nets, where second-order methods are too expensive (O(p^2) in parameters, where p is on the order of millions).
I'm not sure conjugate gradient methods are used much, due to the non-convexity of the merit functions, and hard constraints aren't used much either.
Re:An overview of gradient descent optimization algorithms
by partykid92 on Wednesday November 09, 2016 @04:43PM
(#12914968)
quasi newton methods are not square in the dimension of the problem (think limited memory L-BFGS), and can be run in linear time. In my experience, however, they're 2-3 times slower than regular methods like ADAM.
Useful reference, 6 months old.
Side note, anybody aware of any implementations of the "learning to learn gradient descent by gradient descent" work [0]? I'd love to boost my training time, but I'm worried implementing it myself will just result in an additional set of hyperparameters to tune.
[0] https://arxiv.org/abs/1606.04474
And I saw no mention of constraints.
I'm not sure conjugate gradient methods are used much, due to the non-convexity of the merit functions, and hard constraints aren't used much either.