News for hackers. Stuff that matters.

A comment on Meta's Llama 3.1 can recall 42 percent of the first Harry Potter book. Parent comment

Re:Meta's Llama 3.1 can recall 42 percent of the first Harry... by otabdeveloper4 on Monday June 16, 2025 @04:15AM (#44287573)
LLMs aren't probabilistic. The randomness is bolted on top by the cloud providers as a trick to give them a more humanistic feel.

Under the hood they are 100% deterministic, modulo quantization and rounding errors.

So yes, it is very much possible to use LLMs as a lossy compressed archive for texts.

[ Parent | on HN ]
Re:Meta's Llama 3.1 can recall 42 percent of the first Harry... by fennecfoxy on Monday June 16, 2025 @06:38AM (#44288232)
Has nothing to do with "cloud providers". The randomness is inherent to the sampler, using a sampler that picks top probability for next token would result in lower quality output as I have definitely seen it get stuck in certain endless sequences when doing that.

Ie you get something like "Complete this poem 'over yonder hills I saw' output: a fair maiden with hair of gold like the sun gold like the sun gold like the sun gold like the sun..." etc.

[ Parent | on HN ]
Re:Meta's Llama 3.1 can recall 42 percent of the first Harry... by otabdeveloper4 on Monday June 16, 2025 @07:06AM (#44288415)
> would result in lower quality output

No it wouldn't.

> seen it get stuck in certain endless sequences when doing that

Yes, and infinite loops is just an inherent property of LLMs, like hallucinations.

[ Parent | on HN ]
Re:Meta's Llama 3.1 can recall 42 percent of the first Harry... by fennecfoxy on Thursday June 19, 2025 @10:40AM (#44319125)
How would it not result in lower quality output? You're reducing the set of tokens that may be selected to 1. The pool isn't necessarily synonyms but words that share some semantic connection to the previous word, but the selection of one word in particular can certainly impact the word that is selected next.

Explain your reasoning otherwise.

[ Parent | on HN ]
Re:Meta's Llama 3.1 can recall 42 percent of the first Harry... by otabdeveloper4 on Thursday June 19, 2025 @02:22PM (#44321177)
> You're reducing the set of tokens that may be selected to 1.

Yes, reducing it to 1 token that is deemed to be the optimal token according to the model.

[ Parent | on HN ]