Ask HN: Does Google use the text inside gdocs and Gmail for training AI models?

Re:Ask HN: Does Google use the text inside gdocs and Gmail f... by CrypticShift on Wednesday February 15, 2023 @02:17PM (#34809087)

A related question: are the 40+ [1] millions Full Text Books used ?

OpenAI is using book1 (BookCorpus ?) and book2 sources. By the number of tokens, this seems less than a million book in total.

[1] https://www.blog.google/products/search/15-years-google-book...

[ Parent | on HN ]

Re:Ask HN: Does Google use the text inside gdocs and Gmail f... by jll29 on Wednesday February 15, 2023 @03:18PM (#34809952)

It's "BooksCorpus" (with an 's'), a 800M word dataset described in Zhu et al. (2015) IEEE ICCV, and also available on AWS at: https://aws.amazon.com/marketplace/pp/prodview-d3ghxqzkitn6y

The Google BERT paper (Devlin et al., 2018) also references it: https://aclanthology.org/N19-1423/

Privacy questions aside (as important as they of course are), it's very important to know what a model was trained on exactly: if Wikipedia was used in the training set, you can't use questions from Wikipedia to test it (as that would be cheating) - test data must be as "unseen" as a good exam.

[ Parent | on HN ]

Re:Ask HN: Does Google use the text inside gdocs and Gmail f... by touringa on Wednesday February 15, 2023 @03:24PM (#34810036)

Actually, it's 'BookCorpus'. OpenAI spelt it wrong in their GPT-1 paper.

It has also been analyzed here and here:

https://arxiv.org/abs/2105.05241

https://lifearchitect.ai/whats-in-my-ai/

[ Parent | on HN ]