It's funny but looking at those prediction graphs bouncing up and down like crazy ... one immediately thinks "yep I know, that's exactly what they said would happen if I used polynomials for fitting functions". And yes, that's what polynomials do.

They don't have good local behavior: if a -> f(a) and b -> f(b) are fitted with a higher-degree polynomial then the image of f between a and b will often be infinite (meaning there is some value x between a and b where f(x) is infinite), starting at very high degrees (ie. deep networks) there will probably be many such points.

Intuitively I think of it like this: if you look at the "real world" as a function you can make a couple of very general observations. F(x) -> doesn't have very much information about the world, and it's very hard to make sense of. Most of it just doesn't seem relevant. d/dx F(x) ... much more relevant. d^2/dx F(x) ... also pretty interesting. d^3/dx F(x) less interesting but occasionally important. d^4/dx F(x) ... nobody cares (also if you take camera images and calculate this, it'll be almost exclusively zeroes).

Secondly there are strong "domains" in the real world that we just seem to be unwilling to accept. Polynomials are good in the sense that if you get the equation for a stone dropping onto your foot really, really tighly correctly fitted, that equation holds up for the movement of an entire planet, which is great. But why bother ? It is much more valuable to be able to predict whether a stone will fall on my foot than how Venus will move. That's if you get it right. If you get the polynomial degree of your equation wrong ... it makes utterly ridiculous predictions. Many other approximation methods don't suffer from this problem

Doesn't happen with spline, beziers, even taylor approximations have better behavior.