News for hackers. Stuff that matters.
Re:Why Do Neural Networks Need an Activation Function? by H8crilA on Wednesday July 03, 2019 @02:44AM (#20342157)
Yeah but not for any function, given that f(x)=y, f(2x)=2y. Which is what backpropagation tries to do (find x such that f(x) is something).

Anyhow we're talking past each other. I mean that it's linear in a very strict/specific sense. Take a fixed newtork with a fixed loss function, for a particular set of inputs and particular target outputs (think 1 batch, or even just one example). Now as you backprop you continuously compute derivatives of what's currently on the layer with respect to the loss value, starting from the loss function, then the bottom most layer, then the previous layer and so on.

Now this can be decomposed into a series of matrix multiplication (chain rule), where yes, in any particular run the matrices depend on the weights, inputs and outputs in a nonlinear fashion. But the overall effect is just a composition of linear operators (matrix multiplication). That's why things go to shit when the network is deep and the derivative matrices are all contracting or all expanding.

And just to connect with your example: if f=d(error)/dW then yes d(2error)/dW=2f.

Re:Why Do Neural Networks Need an Activation Function? by p1esk on Wednesday July 03, 2019 @11:51PM (#20351359)
Ok, I see what you mean, and you're right. However, I'm not sure I see any usefulness of this property.
Re:Why Do Neural Networks Need an Activation Function? by H8crilA on Tuesday July 09, 2019 @11:58PM (#20399045)
It helps understand why NaNs or close to zero training speed happens. For example - say you have an LSTM. The more you unroll it the more matrices get multiplied on the way back, and the backprop signal can get amplified/muted in an exponential fashion (as a function of the number of unrolled cells).
Re:Why Do Neural Networks Need an Activation Function? by p1esk on Wednesday July 10, 2019 @12:12AM (#20399087)
I was talking about the property that gradients are linearly proportional to loss value. I don't see whether this property helps or hurts vanishing gradients problem (e.g. if it was not linear, maybe it would be worse).