News for hackers. Stuff that matters.
Re:How LLMs Work, Explained Without Math – Miguel Grinberg by throw310822 on Monday May 06, 2024 @07:00AM (#40273233)
> When the LLM generates each word it does so because it is, on average, the most common word in a corpus of text on which it was trained: "wearing" follows, "I like what you were" because most people who were having these conversations, captured in the training data, were talking about clothes.

Yes. Now extend this to the context length of GPT-4 turbo- about 240 pages of text. So from your description, "wearing" is just the "most common word" to follow those 240 pages of (previously unseen, unique) text according to its training data. Quite simple, nothing to see here, I suppose.

There are no comments on this comment yet.