Sounds like something right up the domain of synthetic data.

There's a sufficient number of old texts - far more than necessary - if we were willing to converse with it in Latin*! Perhaps we could make a translator LLM to converse with the 17th century LLM...

* I'm not talking about Roman-era Latin, Latin was the scholarly/international language for Europe for a long long time. AFAIK, most European Latin literature is untranslated...

Are you sure? In Andrej Karpathy's intro to LLMs[1], he says pre-training uses about 10TB of text from the web.

It's hard to believe that a comparable amount of text had been created, even across all languages, in the entire history of humanity up to the 17th century.

[1] https://www.youtube.com/watch?v=zjkBMFhNj_g

If data quality is good, one can reach results with much less data:

https://arxiv.org/abs/2305.07759

How many stories/tokens did they actually train on? I can't find it in the paper.

Maybe this will help?

https://huggingface.co/datasets/roneneldan/TinyStories

Also note

https://arxiv.org/abs/2309.05463 (which is larger - obviously size still does contribute to performance)

Yes. I think we may have enough for "full finetuning" and erasing to a large extent the previous knowledge. But that's still very far off for pretraining.

"RomeGPT" is next on my list of Monad successors and to give you a general idea, we have on the order of tens of millions of words in classical Latin (and biggest source will… Augustine). There was a BERT Latin project that was able to collect roughly 500 million words in all with mostly early modern and modern Latin.

In comparison I'm currently part of a project to pretrain a French model and we need… 140 billion words.