Re:Collapse of Self-Trained Language Models
by xanderlewis on Wednesday April 17, 2024 @04:20PM
(#40069641)
> ...and then repeat until the model says stop.
How does this actually work? At a high level, the autoregressive token generation process makes sense to me, but how does it know when a sensible time to stop is rather than just going on forever or abruptly stopping after n tokens? Is it trained on text that has special 'stop' tokens inserted at the end of paragraphs, etc. and when it chucks out one of these the model halts?
One of the tokens represents stopping. If you sample stop from the probability distribution instead of a normal text token, then you stop autoregressive sampling.
Re:Collapse of Self-Trained Language Models
by astrange on Wednesday April 17, 2024 @05:40PM
(#40070341)
Open source models have an explicit endoftext token, yes.
You can set llama.cpp to keep generating after that and it often instantly goes offtopic.
Re:Collapse of Self-Trained Language Models
by llm_trw on Wednesday April 17, 2024 @09:44PM
(#40072121)
Not only off topic but out of sample.
The old style LLM hallucinations where it starts spitting out random tokens like:
nxrt$$.full #`,,, jump #43one daisy daisy give me your answer ###
Re:Collapse of Self-Trained Language Models
by gwern on Wednesday April 17, 2024 @04:03PM
(#40069479)
> One of the tricks OpenAI uses when fine-tuning models is to use the unadjusted foundation model as a coherency model alongside the data or reward model to be fine-tuned on. Loss is calculated as the sum[0] of both the fine-tuning and coherency loss so that the training process is anchored to something.
You mean the K-L constraint?
Re:Collapse of Self-Trained Language Models
by tempusalaria on Wednesday April 17, 2024 @04:36PM
(#40069806)
In InstructGPT there is a direct pretraining gradient addition as well as KL penalty
Re:Collapse of Self-Trained Language Models
by corimaith on Thursday April 18, 2024 @12:45AM
(#40073041)
Long Term and Short Term "memory" is a well known concept on NLP, the predecessor to Transforms were Long-Short-Term Mememory networks aka LSTMs. And LSTMs are just RNNs that attempt to solve the vanishing gradient problem. The whole strength of Transformers is that they are capable of processing the entire text corpus in parallel, hence they can "hold" long range dependencies better.
The reason why unanchored training fails is fairly simple. "Training" is a misnomer, we're really copying and compressing[1]. When you train a model on itself, you're making a lossy copy of the original, which isn't a very good truth anchor.
There's probably other ways to anchor a self-training process, though. ChatGPT and other text-to-text transformer models are operated as autoregressive processes, where the model spits out a probability distribution, which you then sample to get a token to add to the input, and then repeat until the model says stop. You'll notice that if you squint a little, this looks like the policy function of AlphaGo, but being run stochastically instead of being min-maxed. Which begs the question: why can't we train GPT like we train chess AI, with self-play followed by fine-tuning on the result, as scored by some kind of reward model?
Granted, you'd have to specify a reward model, as well as what behavior you're trying to 'reward'. One other idea that's been bouncing around my head for self-training is training a model to remember details of prior conversations that have since fell off the end of the context window. The biological analogy being "long-term memory", in contrast to the "short-term memory" of the context window. So perhaps your reward model is the model plus the current context window, and your loss is calculated on the same model but without the parts of the context window you want to free up.
No clue if this has already been done, but if it has please reply with the name of the thing I'm not aware of.
[0] Or difference, I forget. If you get the signs wrong you get a hilariously horny version of ChatGPT.
[1] And, thanks to induction heads, compressing the knowledge of how and what to copy.