Re:Mistral 7B, the most powerful language model for its size...
by brucethemoose2 on Wednesday September 27, 2023 @10:47PM
(#37684576)
All generated output depends on all the previous output. The model "looks" at (mostly) everything every time.
Generation never actually stops, the model just emits a special stop token when stopping is most likely next token. Hence the grammar implementation can prevent this stop token from being emitted prematurely.
There was some discussion of models getting "stuck" where there is no syntactically correct token to emit. Some proposals included a "backspace token" IIRC, but I dunno what they actually did. You can look through the discussion in the PR.
Re:Mistral 7B, the most powerful language model for its size...
by amilios on Friday September 29, 2023 @04:49PM
(#37709929)
Oh yeah that's true! Just block the stop token. But yes, my thought is that there are scenarios where it can get "stuck" as you said. I'll look at the PR, thanks!
Generation never actually stops, the model just emits a special stop token when stopping is most likely next token. Hence the grammar implementation can prevent this stop token from being emitted prematurely.
There was some discussion of models getting "stuck" where there is no syntactically correct token to emit. Some proposals included a "backspace token" IIRC, but I dunno what they actually did. You can look through the discussion in the PR.