News for hackers. Stuff that matters.

A comment on Pause Giant AI Experiments: An Open Letter. Parent comment

Re:Pause Giant AI Experiments: An Open Letter by sarchertech on Wednesday March 29, 2023 @05:13PM (#35363749)
Because they trained it on a huge percentage of the existing web. There's isn't a (much) bigger and better data set available.
[ Parent | on HN ]
Re:Pause Giant AI Experiments: An Open Letter by yreg on Wednesday March 29, 2023 @05:22PM (#35363906)
What percentage?

What about books, newspapers, documents, etc.?

[ Parent | on HN ]
Re:Pause Giant AI Experiments: An Open Letter by JW_00000 on Wednesday March 29, 2023 @05:36PM (#35364090)
The LLaMA paper [1] (Meta's model) contains details about what they trained it on. This includes all of Wikipedia, a huge part of the internet (3.3 TB + 783 GB), a huge set of books (85 GB). My guess is basically all high-quality English articles on the web have been included. Also almost all English books must be included. Newspaper archives is about the only thing I see as missing, as well as more non-English sources.

[1] https://arxiv.org/abs/2302.13971

[ Parent | on HN ]