Re:Tracing the thoughts of a large language model
by bonoboTP on Thursday March 27, 2025 @11:20PM
(#43501162)
This is true, but it doesn't mean that it decided those first 25 without "considering" whether those 25 can be afterwards continued meaningfully with further 25. It does have some internal "lookahead" and generates things that "lead" somewhere. The rhyming example from the article is a great choice to illustrate this.
Like for a 500 token response, at some point it was certain that the first 25 words are the right ones, such that it won't have to take any of them back when eventually calculating the last 25.