Kolmogorov Neural Networks can represent discontinuous functions

Re:Kolmogorov Neural Networks can represent discontinuous fu... by yorwba on Sunday November 05, 2023 @08:52AM (#38151112)

Even in the continuous case, there are nowhere-differentiable continuous functions https://en.wikipedia.org/wiki/Weierstrass_function where attempting to use back-propagation is unlikely to go well.

The paper only gives an existence result for the general case, and may require an arbitrarily complex activation function to represent arbitrarily complex multivariate functions, so it's unlikely to be useful for machine-learning applications.

[ Parent | on HN ]

Re:Kolmogorov Neural Networks can represent discontinuous fu... by ndriscoll on Sunday November 05, 2023 @12:52PM (#38153556)

I can't remember now and never gave measure theory a proper study, but isn't there a happy result that you can quotient out by a space of functions with zero integral to get an almost everywhere differentiable function or something? Maybe that's for L2:continuous, not continuous:differentiable?

Even a result like f(x) = nice(x) + evil(x) with |evil| < epsilon should be "happy enough" right?

[ Parent | on HN ]

Re:Kolmogorov Neural Networks can represent discontinuous fu... by yorwba on Sunday November 05, 2023 @03:04PM (#38155065)

For a "happy enough" approximation to the Weierstrass function, you can cut off the series once a^n/(1 - a) < epsilon and get a function that's differentiable everywhere. But it'll have an extremely large number of local optima, which is still bad for gradient descent.

[ Parent | on HN ]