Re:Why Do Neural Networks Need an Activation Function?
by H8crilA on Tuesday July 09, 2019 @11:58PM
(#20399045)
It helps understand why NaNs or close to zero training speed happens. For example - say you have an LSTM. The more you unroll it the more matrices get multiplied on the way back, and the backprop signal can get amplified/muted in an exponential fashion (as a function of the number of unrolled cells).
[ Parent
| on HN ]
Re:Why Do Neural Networks Need an Activation Function?
by p1esk on Wednesday July 10, 2019 @12:12AM
(#20399087)
I was talking about the property that gradients are linearly proportional to loss value. I don't see whether this property helps or hurts vanishing gradients problem (e.g. if it was not linear, maybe it would be worse).