I've found this fun way to think of it: the goal is to invent a faster form of evolution for pattern recognition, learning, and autonomous task completion. I think one needs to consider it more like biology and a science than pure logic and math. We can discover things that work, and then after that we can study them to learn why they work, just like we don't fully understand the brain yet.

I think there are some really cool problems, such as:

    1. Is synthetic data viable for training?
    2. How do you make deep learning agents that can do task planning and introspection in complex environments?
    3. How do we efficiently build memory and data lookup into AI agents? And is this better/worse than making longer context windows?

Are deep learning and neural networks just curve fitting? I thought those were significantly different.

You could argue all the building blocks are forms of curve fits, but that isn't a terribly useful statement even if true. If you can fit a curve to the desired behavior of any function, or composition of functions (which is a function) then you can solve any problem you can express the desired behavior of. Including the expressing of desired behavior for some other class if problems. Saying it is just curve fitting is like saying something is just math. The entirety of reality is just math.

By that logic, anything that is predictive is curve fitting, including entire academic fields like physics and climatology. You could say that all automation is curve fitting. I don’t think there’s much to be gained by being that reductive.

From a technical standpoint, it’s not correct analogy either, because it assumes you have a curve to fit. What curve is language? What’s curve is images? No answer, because there isn’t one. Deep learning is about modeling complex behaviors, not curve fitting. Images and language for instance are based in social and cultural patterns and not intrinsic curves to be fit.

At best, it’s an imprecise statement. But I’d disagree entirely.

Although it fundamentally is curve fitting, I'd venture to say that at some point, having to handle millions of parameters makes the curve fitting problem unrecognizable... A change in quantity is a change is nature if you will.

IOW: to me, fitting a generalized linear model is very different than fitting a convolutional network.