Every model learned by gradient descent is approximately a kernel machine (2020)

Re:Every model learned by gradient descent is approximately ... by sushibowl on Sunday February 25, 2024 @07:06AM (#39500126)

> Surely learning an algorithm is memorization?

Not in this context, no. An LLM is never given any algorithm to memorise. It is given only input->output sets, and it "learns" what the algorithm is from those sets.

We know that it does this because it is able to generalize that algorithm to inputs and outputs outside of its set of training examples. So we know that it doesn't only memorise which input connects to which output, and regurgitate that information. It has come to "understand" the formula that connects the input to the output, without ever being given that formula.

[ Parent | on HN ]

Re:Every model learned by gradient descent is approximately ... by quantum_state on Sunday February 25, 2024 @08:15AM (#39500549)

It’s fitting a function to data and guessing the value of the function for a new input. It does not know what is under the hood of the function, etc., and see the implications when the function produces a wrong value … etc.

[ Parent | on HN ]

Re:Every model learned by gradient descent is approximately ... by astrange on Sunday February 25, 2024 @09:09PM (#39506942)

It can do that. Verifying an answer is just another algorithm it can learn.

LLMs mostly can't do math but that, like most of their other flaws, is because of the tokenizer.

[ Parent | on HN ]