Ah so it is! I guess my point was that even this simple function is enough, you don't need any 'magic' beyond that.

Well, yes. The 'magic' is the nonlinearity. It's because the composition of linear (actually also affine) functions is still just linear (affine). You don't get any additional power by combining many of them - which is also the reason why linear functions are so well understood and easy to work with.

You give sprinkle in a tiny nonlinearity (e.g. x^2 instead of x) and suddenly you can get infinite complexity by weighted composition - which is also the reason why we're so helpless with nonlinear functions and reach for linear approximations immediately (cf. gradient descent).

>It's because the composition of linear (actually also affine) functions is still just linear (affine).

Except you can "compose" affine functions using Horner's schema to get any polynomial... No need to sprinkle tiny non linearities. It's a buffet. Grab as much as you need.

So is IEEE floating point https://www.youtube.com/watch?v=Ae9EKCyI1xU