News for hackers. Stuff that matters.

A comment on TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy. Parent comment

Re:TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy by atq2119 on Monday February 10, 2025 @04:36AM (#42998561)
Perhaps that's not reasonable to expect, but Meta apparently kind of did it anyway, if not in a way that helps reproduce their LLM: https://arstechnica.com/tech-policy/2025/02/meta-torrented-o...
[ Parent | on HN ]
Re:TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy by moffkalast on Monday February 10, 2025 @05:18AM (#42998809)
Actually they did, the entire 15T tokens that were supposedly used for training the llama-3 base models are up on HF as a dataset: https://huggingface.co/datasets/HuggingFaceFW/fineweb

It's just not literally labelled so because of obvious reasons.

[ Parent | on HN ]