News for hackers. Stuff that matters.
Re:“Deep Learning has outlived its usefulness as a buzz-phrase” by fwilliams on Saturday January 06, 2018 @01:39AM (#16084164)
There is literature on Quasi-Newton and Krylov Subspace methods for training Neural Networks. For example, https://dl.acm.org/citation.cfm?id=3104516.

I think the primary reason that such methods are not used much in practice is memory and computational cost: each function evaluation is expensive and you need to solve a very large system at every iteration.

Also to reply to a sibling comment, you can add momentum and step length adjustments to second-order methods in much the same way as in steepest-descent to help escape saddles. The only difference is how the descent direction is chosen for the optimization.

Re:“Deep Learning has outlived its usefulness as a buzz-phrase” by steev on Saturday January 06, 2018 @08:15AM (#16085128)
This is correct - second order methods are great in theory, but they are generally computationally prohibitive for high dimensional problems.