Re:We need a much more sophisticated debate about AI
by HarHarVeryFunny on Tuesday April 04, 2023 @09:44PM
(#35448185)
I'm not sure that this apple/pear example proves anything. In the context of "apple tree and picked" in the input, it's enough simply that the word "an" received positive feedback during training as the next word, and having generated "an" then "apple" becomes almost certain as the next word after that.
For sure there has to be some internal high level representation of the entire input, and I would expect also some representation of possible continuations at a higher level than just alternate word choices, but I wouldn't exactly characterize this as a plan since it's so statistical in nature, and the model has to continually adapt to what is actually generated (sampled).
At a low temperature setting "an" might always be picked, making "apple" more likely to follow, but at a higher temperature maybe another word such as "the" might be chosen, perhaps with a continuation of "biggest apple i could find", etc. In a 350B token training set I'm sure there are a number of other alternatives too.
So, does the model plan ahead ? Well, sort of, but this "plan" has to represent multiple branching alternate futures, and continually adapt as the output is sampled, so it's a very fluid kind of plan. It's also not clear how far ahead the plan needs to represent .. maybe one word only (like a state machine), even if the reason those alternate one-word continuations are there is because they were seen as part of specific longer continuations.
The lack of any internal state preservation from one word generated to the next puts a huge limit on what can be planned, as does the extent of how complex a plan can be generated from the flow of a single input sequence forwards thru the layers of the model. You can see the limitations of this with prompts such as "pick two primes > 200, and tell me their product, but not the numbers themselves"... if you play with it and analyze the mistakes, it makes it clear how much the output really is generated word by word, adapting to what came before, much of the time.
For sure there has to be some internal high level representation of the entire input, and I would expect also some representation of possible continuations at a higher level than just alternate word choices, but I wouldn't exactly characterize this as a plan since it's so statistical in nature, and the model has to continually adapt to what is actually generated (sampled).
At a low temperature setting "an" might always be picked, making "apple" more likely to follow, but at a higher temperature maybe another word such as "the" might be chosen, perhaps with a continuation of "biggest apple i could find", etc. In a 350B token training set I'm sure there are a number of other alternatives too.
So, does the model plan ahead ? Well, sort of, but this "plan" has to represent multiple branching alternate futures, and continually adapt as the output is sampled, so it's a very fluid kind of plan. It's also not clear how far ahead the plan needs to represent .. maybe one word only (like a state machine), even if the reason those alternate one-word continuations are there is because they were seen as part of specific longer continuations.
The lack of any internal state preservation from one word generated to the next puts a huge limit on what can be planned, as does the extent of how complex a plan can be generated from the flow of a single input sequence forwards thru the layers of the model. You can see the limitations of this with prompts such as "pick two primes > 200, and tell me their product, but not the numbers themselves"... if you play with it and analyze the mistakes, it makes it clear how much the output really is generated word by word, adapting to what came before, much of the time.