News for hackers. Stuff that matters.
Re:Everything Is a Linear Model by EvgeniyZh on Sunday February 18, 2024 @11:15PM (#39426177)
"You can make linear model non linear by adding non linearity" is kind of vacuous statement, isn't it
Re:Everything Is a Linear Model by ivan_ah on Sunday February 18, 2024 @11:44PM (#39426305)
No no, linear just means some expression of the form

    y = b1*x1 + b2*x2 + ... + bp*xp
which is in the heart of the model (perhaps more specific description would be linear combination). Whatever you call it, it's just some quantity y, that is constructed via an additive process from components x1...xp, and each component is multiplied by some constant (the coefficient of the linear combination).

This linear combination "core" of the model can be used directly (in which case it is geometrically a hyperplane), but you could also use non-linear transformation of the inputs or the outputs.

E.g. x1 could be a spline basis function, or log-transformed data, or anything.

Similarly, the output y can be passed though some non-linearity to force a particular output (like in logistic regression). Hence the statement is not vacuous

Re:Everything Is a Linear Model by EvgeniyZh on Monday February 19, 2024 @01:53AM (#39426854)
Well yeah, but what is the non trivial statement here? Linear model is linear, and if you add to your model non-linear steps (such as pre- or post- processing) then it is not longer linear and can handle non-linear data. In the same spirit, you could say that GPT4 is a linear model (token logits) with a very large preprocessing (transformer).

Maybe what OP meant is that particular non linearities (splines) allow to keep some goodies from the linear model in non-linear settings. Still it's not clear for me in which settings exactly such models are good enough