You can find GPT-2's training dataset list - at a high level - in the GPT-2 repository on Github: https://github.com/openai/gpt-2/blob/master/model_card.md#da... However, OpenAI goes dark after that regarding the 'data soup' that was fed into their LLMs. In general, start around 2019 and definitely by 2020 you'll notice that research labs became much less forthcoming about the data that went into their models. As far as I'm aware, BookCorpus is one of the more commonly-used 'large books dataset' that's been utilized in recent years to train large language models (LLMs) like generative pretrained transformers: https://12ft.io/proxy?q=https%3A%2F%2Ftowardsdatascience.com...

At my alma mater I remember the large-scale Google book scanning devices and what a herculean effort that was to digitize the largest university library system's books - University of Michigan - although only 7M texts from the entire collection of ~16 million texts: https://en.wikipedia.org/wiki/University_of_Michigan_Library) were digitized.I too was curious about the state of the Google Books project: https://www.edsurge.com/news/2017-08-10-what-happened-to-goo...

This is an interesting piece of ephemera from 2005, when Google started digitizing books at UMich: https://apps.lib.umich.edu/files/services/mdp/faq.pdf

As far as I recall, the Books project allowed the early n-grams functionality to be built out: https://ai.googleblog.com/2006/08/all-our-n-gram-are-belong-...

The Google Books Ngram Viewer tool is actually still in existence; you can play around with it here: https://books.google.com/ngrams/graph?corpus=0&content=Vorsp...

If that was the case, it threw more than half of it up again, because it's not making much sense atm.

Yes, and while there were copyright issues with them putting the books out there in public, they still retain all the scans to use for search projects.

https://books.google.com/

I'd be crazy if I didnt think that google is sitting on some stuff that nobody knows about and they are stroking their cat from the lair as we type.

"...and they are stroking their cat from the lair..."

On the first quick read though, I thought to myself, "Can he use that sort of language here?"

Then I pictured Dr. Evil and it made more sense...

I think Blofeld was the reference. Dr Evil is a parody of Blofeld.

It’s funny that the general internet pessimism about Google misses stuff like this.

I mean ChatGPT 3 went viral and Google managed to ship Bard in a few weeks. I think the consensus is that ChatGPT is better, but it was literally sitting on the shelf ready to go.

The cat has been deprecated half a year ago. ;-)

It was claimed to use book data, but IMHO nowadays the available internet data is larger than all the books ever published; so while book data definitely should be used, it's not a pathway to significant increases in data size.