Re:TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy
by a2128 on Monday February 10, 2025 @03:13AM
(#42997978)
Is it reasonable to expect companies to redistribute 100TB of copyrighted content they used for their LLM, just on the off-chance someone has a few million laying around and wants to reproduce the model from scratch?
[ Parent
| on HN ]
Re:TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy
by prisenco on Monday February 10, 2025 @03:40AM
(#42998101)
Would a reference file with filename, size, source and checksum count towards the OSI definition?
Re:TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy
by anotherhue on Monday February 10, 2025 @03:15AM
(#42997990)
The RL-only (no SFT) approaches might remove that issue. Problem sets should be smaller (and mechanically creatable) than the entire western corpus.
Re:TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy
by puapuapuq on Monday February 10, 2025 @03:34AM
(#42998071)
For open source model claiming SOTA performance, we could at least check for data leak from its training data.
Re:TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy
by atq2119 on Monday February 10, 2025 @04:36AM
(#42998561)
Perhaps that's not reasonable to expect, but Meta apparently kind of did it anyway, if not in a way that helps reproduce their LLM: https://arstechnica.com/tech-policy/2025/02/meta-torrented-o...
Re:TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy
by moffkalast on Monday February 10, 2025 @05:18AM
(#42998809)
Actually they did, the entire 15T tokens that were supposedly used for training the llama-3 base models are up on HF as a dataset: https://huggingface.co/datasets/HuggingFaceFW/fineweb
It's just not literally labelled so because of obvious reasons.
Re:TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy
by TeMPOraL on Monday February 10, 2025 @05:55AM
(#42999001)
Redistribute? No. Itemize and link to? Yes.
With LLMs, the list doesn't even have to be kept up to date, nor the links alive (though publishing content hashes would go a long way here). It's not like you can get an identical copy of a model built anyway, there's too much randomness at every stage in the process. But, as long as the details of cleanup and training are also open, a list of training material used would suffice - people would fetch parts of it, substitute other parts with equivalents that are open/unlicensed/available, add new sources of their own, and the resulting model should have similar characteristics to the OG one who we could, now, call "open source".