Meta's Llama 3.1 can recall 42 percent of the first Harry Potter book

Re:Meta's Llama 3.1 can recall 42 percent of the first Harry... by fennecfoxy on Monday June 16, 2025 @06:38AM (#44288232)

Has nothing to do with "cloud providers". The randomness is inherent to the sampler, using a sampler that picks top probability for next token would result in lower quality output as I have definitely seen it get stuck in certain endless sequences when doing that.

Ie you get something like "Complete this poem 'over yonder hills I saw' output: a fair maiden with hair of gold like the sun gold like the sun gold like the sun gold like the sun..." etc.

[ Parent | on HN ]

Re:Meta's Llama 3.1 can recall 42 percent of the first Harry... by otabdeveloper4 on Monday June 16, 2025 @07:06AM (#44288415)

> would result in lower quality output

No it wouldn't.

> seen it get stuck in certain endless sequences when doing that

Yes, and infinite loops is just an inherent property of LLMs, like hallucinations.

[ Parent | on HN ]

Re:Meta's Llama 3.1 can recall 42 percent of the first Harry... by fennecfoxy on Thursday June 19, 2025 @10:40AM (#44319125)

How would it not result in lower quality output? You're reducing the set of tokens that may be selected to 1. The pool isn't necessarily synonyms but words that share some semantic connection to the previous word, but the selection of one word in particular can certainly impact the word that is selected next.

Explain your reasoning otherwise.

[ Parent | on HN ]

Re:Meta's Llama 3.1 can recall 42 percent of the first Harry... by otabdeveloper4 on Thursday June 19, 2025 @02:22PM (#44321177)

> You're reducing the set of tokens that may be selected to 1.

Yes, reducing it to 1 token that is deemed to be the optimal token according to the model.

[ Parent | on HN ]