News for hackers. Stuff that matters.
Re:My Python code is a neural network by jb1991 on Monday July 01, 2024 @11:12AM (#40846565)
FYI, there are actually many algorithms going back longer than the neural network algorithm that have been proven to be a universal function approximator. Neural networks are certainly not the only and not the first to do so. There are quite a few that are actually much more appropriate for many cases than a neural network.
Re:My Python code is a neural network by derangedHorse on Monday July 01, 2024 @11:42AM (#40846890)
What other algorithms can do this and which situations would they be more useful than neural networks?
Re:My Python code is a neural network by gnyeki on Monday July 01, 2024 @12:03PM (#40847107)
This area is covered by non-parametric statistics more generally. There are many other methods to non-parametrically estimate functions (that satisfy some regularity conditions). Tree-based methods are one family of such methods, and the consensus still seems to be that they perform better than neural networks on tabular data. For example:

https://arxiv.org/abs/2106.03253

Re:My Python code is a neural network by someoneontenet on Monday July 01, 2024 @12:25PM (#40847342)
Newtons Method approximates square roots. Its useful if you want to approximate something like that without pulling in the computational power required of NN.
Re:My Python code is a neural network by astrobe_ on Monday July 01, 2024 @01:27PM (#40848040)
I think the problem to solve is more like : given a set of inputs and outputs, find a function that gives the expected output for each input [1]. This is like Newton's method on a higher order ;-). One can find such a tool in Squeak or Pharo Smalltalk, IIRC.

[1] https://stackoverflow.com/questions/1539286/create-a-functio...

Re:My Python code is a neural network by kristjansson on Monday July 01, 2024 @01:12PM (#40847888)
Newton's method related to universal function approximation in the same way a natural oil seep is related to a modern IC engine...
Re:My Python code is a neural network by jb1991 on Tuesday July 02, 2024 @02:46AM (#40854121)
By definition, that’s not a “universal“ function approximator.
Re:My Python code is a neural network by jb1991 on Monday July 01, 2024 @03:24PM (#40849547)
Re:My Python code is a neural network by dontlikeyoueith on Monday July 01, 2024 @08:42PM (#40852439)
The Taylor Series dates to 1715. Fourier Series dates to the 1820s.

Both are universal function approximators and both can be learned via gradient descent.

For the case where the function you want to learn actually is polynomial or periodic (respectively), these are better than neural networks.

Re:My Python code is a neural network by camjw on Tuesday July 02, 2024 @07:22AM (#40855535)
For your interest, Taylor Series are not universal function approximators - the Taylor Series around 0 for

f(x) = e^(-1/x^2) if x != 0 else 0

is identically zero (all partial derivatives are 0 at 0) but the function is clearly not identically zero. So the radius of convergence for this Taylor series is infinite but it only equals the approximated function at one point.

I'm sure there are some conditions you can put on f to make the Taylor Series a UFA but it's been quite a while since I did any real analysis so I have forgotten!

Doesn't detract from the overall point though that there are UFAs that are not neural nets. I should say that I don't know what the precise definition of a UFA really is, but I assume you have to have more than equality at one point.

Re:My Python code is a neural network by Horffupolde on Tuesday July 02, 2024 @08:17AM (#40855891)
Taylor series work on differentiable intervals. You specifically chose a function and interval where this is not true. Of course it will not be a good approximation.
Re:My Python code is a neural network by dontlikeyoueith on Tuesday July 02, 2024 @02:02PM (#40858999)
I'm pretty sure the UFA theorems for neural networks wouldn't apply to that function either: https://en.wikipedia.org/wiki/Universal_approximation_theore...

Generally, they assume the function to be approximated is continuous.