Not one which can easily be represented as an interpolation of other tasks it was trained on.

Well, it can't, by definition.

Almost nothing I prompt into existence already exists, otherwise I'd just use the existing thing.

What definition is that?

model

The definitions behind the math of LLMs.

This isn't true.

A neural network can do new things outside the training data. Random initialization sees to that.

Whether those things are useful is a different question, but it turns out that yes, some of those things are useful. It also turns out that in a large-enough LLM if you give it some kind of idea of what you see as useful then yes, it can pick out the newly useful things.

Random init only exists to make each neuron distinct from other neurons. The weight is still defined by the training data, without the training data, these random weight don't give us anything.

The second thing you said is basically what's called fine-tuning.

Yes you are right about random initilization of course. Long day here.

For the second: No, not fine tuning at all.

For one thing in-context learning is enough for learning novel tasks[1].

Secondly, it can do new things outside that data. LLMs build models of the world[2]. That means it can model novel behavior based on that internal representation.

[1] https://arxiv.org/abs/2005.14165

[2] https://arxiv.org/abs/2305.11169

Not sure what you're getting at, in any of the cases you said it needs to learn if it needs to do new things.

Maybe it can generalize outside of some learning task you gave it, but still it comes from the training data.

"Generalization" also comes with some luck, since these models most of the time output complete non-sense when it comes to tasks it "didn't learn" specifically. Just because you got lucky with the neuron connections doesn't mean it has what it takes to generalize anything.

I don't think it can whistle while chewing gum, no.

I think it can patch together components of tasks represented in its training data to do demonstrably new things, yes, which does count as "perform tasks not in its training data" with the limitation that some form of the task must be present. Example: writing an ode to a sales lead for a job well done in 2032 for the first quarter. Odes exist, the concept of sales performance exist, good job performance, and notions of time. Stringing those together are good examples of interpolation but not extrapolation, as it were.

In my opinion that's very handwavey.

Definitely see where you might feel so. I provided an analogy that may help anchor - why ML fails extrapolation is left as an exercise to the determined reader! :)

The surface for "interpolation" is fairly broad for LLMs but it isn't infinite -- extrapolation still exists!

the most obvious example was counting the Rs in strawberry.

figuring out what is and isnt in distribution is quite difficult though given that theres so much data passed in, and transformers have some amount of combining it together

theres some really obvious things though, like picking nonsense tokens together, or things outside of the tokenspace entirely

you could also construct a string that is a series of the least likely tokens in a row. you wont get it to output the next token, despite it being quite clear how you did it