That doesn't mean it knows "in advance" what it want to say, it's just that at every step the alligator is lurking in the logits because it directly derives from the prompt.
It will also emit "a" from time to time without issue though, but will never spit "alligator" right after that, that's it.
> Sure, it derives from the prompt but so does every an LLM generates, and same for any other AI mechanism for generating answers.
Not really, because of the autoregressive nature of LLMs, the longer the response the more it will depend on its own response rather than the prompt. That's why you can see totally opposite response from LLM to the same query if you aren't asking basic factual questions. I saw a tool on reddit a few month ago that allowed you to see which words in the generation where the most “opinionated” (where the sampler had to chose between alternative words that were close in probability) and where it was easy to see that you could dramatically affect the result by just changing certain words.
> "an" gets a high probability because the model internally knows that "alligator" is the correct thing after that.
This is true, though it only works with this kind of prompt because the output of the LLM has little impact on the generation.
Globally I see what you mean, and I don't disagree with you, but at the same time, I think that saying that LLMs have a sense of anticipating the further token misses their ability to get driven astray by their own output: they have some information that will affect further tokens but any token that get spit can, and will, change that information in a way that can dramatically change the “plans”. And that's why I think using trivial questions isn't a good illustration, because it pushes this effect under the rug.
"The animal most similar to a crocodile is:"
https://chatgpt.com/share/67d493c2-f28c-8010-82f7-0b60117ab2...
It will always say "an alligator". It chooses "an" because somewhere in the next word predictor it has already figured out that it wants to say alligator when it chooses "an".
If you ask the question the other way around, it will always answer "a crocodile" for the same reason.