One of the tokens represents stopping. If you sample stop from the probability distribution instead of a normal text token, then you stop autoregressive sampling.
Re:Collapse of Self-Trained Language Models
by astrange on Wednesday April 17, 2024 @05:40PM
(#40070341)
Open source models have an explicit endoftext token, yes.
You can set llama.cpp to keep generating after that and it often instantly goes offtopic.
How does this actually work? At a high level, the autoregressive token generation process makes sense to me, but how does it know when a sensible time to stop is rather than just going on forever or abruptly stopping after n tokens? Is it trained on text that has special 'stop' tokens inserted at the end of paragraphs, etc. and when it chucks out one of these the model halts?