Meta's Llama 3.1 can recall 42 percent of the first Harry Potter book

Re:Meta's Llama 3.1 can recall 42 percent of the first Harry... by aschobel on Monday June 16, 2025 @01:27AM (#44286838)

Indeed! It is a form of massive lossy compression.

> Llama 3 70B was trained on 15 trillion tokens

That's roughly a 200x "compression" ration; compared to 3-7x for tradtional lossless text compression like bzip and friends.

LLM don't just compress, they generalize. If they could only recite Harry Potter perfectly but couldn’t write code or explain math, they wouldn’t be very useful.

[ Parent | on HN ]

Re:Meta's Llama 3.1 can recall 42 percent of the first Harry... by amlib on Monday June 16, 2025 @05:15PM (#44293468)

But LLMs cant write code nor explain math, they only plagiarize existing code and plagiarize existing explanations of math.

[ Parent | on HN ]