Books in .txt format for AI training purposes

Re:Books in .txt format for AI training purposes by mdifrgechd on Sunday October 25, 2020 @11:50AM (#24887170)

I can imagine building a model that is a derived work, but I don't see why all models trained on a corpus that contains some text can automatically be thought of as derived from that text in a creative sense. Models like GPT are using the text as a specific instance of the latent relationships that make up our language. The text is an example of english to learn from, it's not a creative work being extended.

Training a language model, I would argue, is equivalent (in terms of being derivative) to generating a list of the top ten words in a corpus of text. Not really a derived work.

[ Parent | on HN ]

Re:Books in .txt format for AI training purposes by mdifrgechd on Sunday October 25, 2020 @11:56AM (#24887227)

One possible counterpoint- sample based explanation techniques can tell what training data was most influential in a ML model's prediction, and this has been considered in language models[0]. So you could argue that if there are training examples within a corpus that are having an outsized influence on the model output, then maybe it is derivative. This would be pretty cool to look at - are some GPT or other language model outputs relying strongly on a few sources?

[0] https://arxiv.org/abs/1810.03611

[ Parent | on HN ]