News for hackers. Stuff that matters.
Re:Why Do Neural Networks Need an Activation Function? by H8crilA on Tuesday July 09, 2019 @11:58PM (#20399045)
It helps understand why NaNs or close to zero training speed happens. For example - say you have an LSTM. The more you unroll it the more matrices get multiplied on the way back, and the backprop signal can get amplified/muted in an exponential fashion (as a function of the number of unrolled cells).
Re:Why Do Neural Networks Need an Activation Function? by p1esk on Wednesday July 10, 2019 @12:12AM (#20399087)
I was talking about the property that gradients are linearly proportional to loss value. I don't see whether this property helps or hurts vanishing gradients problem (e.g. if it was not linear, maybe it would be worse).