Re:Kolmogorov Neural Networks can represent discontinuous fu...
by magicalhippo on Sunday November 05, 2023 @10:10AM
(#38151768)
This isn't my field at all, so this might be silly.
I was thinking about some weird activation function like say
0 for x < 0.5
0.5 for x in [0.5, 1]
0.1x + 1 for x > 1
While it is piece-wise differentiable, similar to ReLU, I'm guessing regular backpropagation would have would struggle with it.
My probably silly thought was, what if you used a smoothed version for computing the differentials for the backpropagation step, while keeping the discontinuous function for actual evaluation? My thought was this would make backprop sensitive to the step changes in the function, while allowing for discontinuous activation.
Of course this example function is just something random without any further thought, so probably not a useful one in actual usage.
I was thinking about some weird activation function like say
While it is piece-wise differentiable, similar to ReLU, I'm guessing regular backpropagation would have would struggle with it.My probably silly thought was, what if you used a smoothed version for computing the differentials for the backpropagation step, while keeping the discontinuous function for actual evaluation? My thought was this would make backprop sensitive to the step changes in the function, while allowing for discontinuous activation.
Of course this example function is just something random without any further thought, so probably not a useful one in actual usage.