Widely known in the controls engineering world for the last hundred years as 'anything is linear if you zoom in far enough'

Well, subject to certain constraints.

You aren't guaranteed that your equilibria behave the same in the linearization of a nonlinear system if your Jacobian has any eigenvalues with real part of 0.

https://en.wikipedia.org/wiki/Hartman%E2%80%93Grobman_theore...

It's amazing how many ideas I've seen being "rediscovered" these days from control theory.

Wait isn't that just an application of Taylor expansion which analysis has known about for several hundreds of years?

Taylor is only for some functions, must be continuos and differentiable to n grades. In control theory you break the functions, so discontinuities can also be considered.

Alternatively, anything is linear if you add enough dimensions.

In the 70's in CS grad school at USC, I wrote an adaptive least squares cubic spline fit routine. Kept subdividing intervals and fitting a least squares cubic spline in each interval until a criteria was met.

Fun times.

Don't have the code or writeup any more.

This sounds really cool but was hard to digest for me as a ML Engineer who came into work just around deep learning and DNNs.

Is there some go-to practice material I could look at?

Splines I haven't touched since numerical computing exercises in school.

If you know DNNs, you'll find the introduction of splines in Semiparametric Regression in R very intuitive - splines are introduced using what the authors call a "truncated line basis" but you already know it as the RELU function, just with a bias. Indeed, even the penalization of splines will look extremely familiar: it's basically just L2 regularization to induce smoothness.

You might also enjoy reading the "Neural Additive Model" paper from Hinton's lab, which is basically GAMs using a separate DNN as a "spline basis" for each input variable.

Simon Wood's Generalized Additive Model book.

Or, for a more approachable treatment, Semiparametric Regression with R by Harezlak, Ruppert, and Wand. A middle ground between Wood's book (which is comprehensive but can dip into math that's way over my head at times) and H/R/W is Semiparametric Regression by Ruppert, Wand, and Carroll.

I have also heard great things about Frank Harrell's Regression Modeling Strategies which uses a slightly different approach (still spline-based though), but I haven't read it. His other writing is fantastic though.

> Frank Harrell's Regression Modeling Strategies which uses a slightly different approach (still spline-based though), but I haven't read it.

Very very good indeed, with the exception that he basically ignores compute time and efficiency. I learned a bunch, but applying his approach to the kinds of datasets I deal with (much larger and with pretty strict compute budgets) was very difficult.

"You can make linear model non linear by adding non linearity" is kind of vacuous statement, isn't it

No no, linear just means some expression of the form

    y = b1*x1 + b2*x2 + ... + bp*xp

which is in the heart of the model (perhaps more specific description would be linear combination). Whatever you call it, it's just some quantity y, that is constructed via an additive process from components x1...xp, and each component is multiplied by some constant (the coefficient of the linear combination).

This linear combination "core" of the model can be used directly (in which case it is geometrically a hyperplane), but you could also use non-linear transformation of the inputs or the outputs.

E.g. x1 could be a spline basis function, or log-transformed data, or anything.

Similarly, the output y can be passed though some non-linearity to force a particular output (like in logistic regression). Hence the statement is not vacuous

Well yeah, but what is the non trivial statement here? Linear model is linear, and if you add to your model non-linear steps (such as pre- or post- processing) then it is not longer linear and can handle non-linear data. In the same spirit, you could say that GPT4 is a linear model (token logits) with a very large preprocessing (transformer).

Maybe what OP meant is that particular non linearities (splines) allow to keep some goodies from the linear model in non-linear settings. Still it's not clear for me in which settings exactly such models are good enough

> Splines can easily take care of that.

within set boundaries.

GLMs are a non linear transformation on an output followed by linear modelling. They are referred to as “linear models” but you might as well then also consider NNs as linear models, or any model at all which ends with addition as the final step.

Can you recommend some resources to go from step 3 to 4?

So far I have been successfully using GLMMs where appropriate, but then jumped to implementing completely arbitrary models by fitting them to the data (plus bootstrapping).

But that takes ages for most problems as you say