Kolmogorov Neural Networks can represent discontinuous functions

Re:Kolmogorov Neural Networks can represent discontinuous fu... by ubj on Sunday November 05, 2023 @09:42AM (#38151526)

The paper only gives a representation result, so no method is known yet for constructing the component functions `g` for the discontinuous case. See the last paragraph of the Conclusion.

It may turn out that the correct representation can be constructed using an alternate method than backpropagation. But this is still an open question.

[ Parent | on HN ]

Re:Kolmogorov Neural Networks can represent discontinuous fu... by magicalhippo on Sunday November 05, 2023 @10:10AM (#38151768)

This isn't my field at all, so this might be silly.

I was thinking about some weird activation function like say

    0 for x < 0.5
    0.5 for x in [0.5, 1]
    0.1x + 1 for x > 1

While it is piece-wise differentiable, similar to ReLU, I'm guessing regular backpropagation would have would struggle with it.

My probably silly thought was, what if you used a smoothed version for computing the differentials for the backpropagation step, while keeping the discontinuous function for actual evaluation? My thought was this would make backprop sensitive to the step changes in the function, while allowing for discontinuous activation.

Of course this example function is just something random without any further thought, so probably not a useful one in actual usage.

[ Parent | on HN ]