Re:Understanding LSTM networks
by nabla9 on Friday August 28, 2015 @03:37AM
(#10134127)
>If I didn't know that neural nets worked, and someone explained the idea, I'd strongly expect them to get stuck in local minima
This has been the general consensus (based on intuition) until recently. It turns out that it's not as big problem as tought. Almost all local minima have very similar error values. The bigger issue is combinatorially large number of saddle points where gradient becomes zero with very few downward curving directions. Saddle-free Newton method is one way around that.
This has been the general consensus (based on intuition) until recently. It turns out that it's not as big problem as tought. Almost all local minima have very similar error values. The bigger issue is combinatorially large number of saddle points where gradient becomes zero with very few downward curving directions. Saddle-free Newton method is one way around that.
Open Problem: The landscape of the loss surfaces of multilayer networks, Choromanska, LeCun, Arous http://jmlr.org/proceedings/papers/v40/Choromanska15.pdf