News for hackers. Stuff that matters.

A comment on MonadGPT – What would have happened if ChatGPT was invented in the 17th century?. Parent comment

Re:MonadGPT – What would have happened if ChatGPT was invent... by jstanley on Sunday November 26, 2023 @05:05AM (#38420488)
Are you sure? In Andrej Karpathy's intro to LLMs[1], he says pre-training uses about 10TB of text from the web.

It's hard to believe that a comparable amount of text had been created, even across all languages, in the entire history of humanity up to the 17th century.

[1] https://www.youtube.com/watch?v=zjkBMFhNj_g

[ Parent | on HN ]
Re:MonadGPT – What would have happened if ChatGPT was invent... by yyyk on Sunday November 26, 2023 @05:35AM (#38420613)
If data quality is good, one can reach results with much less data:

https://arxiv.org/abs/2305.07759

[ Parent | on HN ]
Re:MonadGPT – What would have happened if ChatGPT was invent... by thomasahle on Sunday November 26, 2023 @10:42PM (#38427784)
How many stories/tokens did they actually train on? I can't find it in the paper.
[ Parent | on HN ]
Re:MonadGPT – What would have happened if ChatGPT was invent... by yyyk on Monday November 27, 2023 @04:12AM (#38430019)
Maybe this will help?

https://huggingface.co/datasets/roneneldan/TinyStories

Also note

https://arxiv.org/abs/2309.05463 (which is larger - obviously size still does contribute to performance)

[ Parent | on HN ]