Re:Kolmogorov Neural Networks can represent discontinuous fu...
by amelius on Sunday November 05, 2023 @10:36AM
(#38152016)
You can also extend the derivative such that you compute it numerically and then some slight discontinuities are certainly not a problem.
Re:Kolmogorov Neural Networks can represent discontinuous fu...
by PeterisP on Sunday November 05, 2023 @11:49AM
(#38152826)
One of the common - if not the most common - activation function is 'rectified linear unit' (ReLU) which is a fancy name for y=max(x,0), which has a discontinuous gradient (1 if x>0, 0 if x<0) and that works mostly fine.
That being said, if it's only discontinuous in a few number of places you could extend the derivative everywhere by taking either the left or the right derivative, and then you'd end up with a gradient being defined everywhere, but not continuous. But then does gradient descent work if the gradient isn't continuous?