News for hackers. Stuff that matters.
Re:OpenAI's justification for why training data is fair use,... by troupe on Thursday October 05, 2023 @01:57PM (#37781781)
> "Fair use" doesn't generally include taking the author's entire work (or for that matter, his entire lifetime's corpus).

If someone studies every one of Stephen King's books, studies King's style, studies his plots, etc. and uses the knowledge to learn how to be a writer, there is no need to invoke "fair use" unless they are actually copying King's books. Studying an author's corpus in order to be able to produce works like the author is not copyright infringement. It is just the way we learn...and the way LLMs learn as well.

If we want to say it isn't allowed because LLM read too quickly, then we could do that. Maybe make them not read any faster than humans. But LLMs aren't doing something different than humans other than just the speed at which they do it.

Re:OpenAI's justification for why training data is fair use,... by ang_cire on Thursday October 05, 2023 @02:08PM (#37781927)
If you read the PDF, you'd know that OpenAI admits they ARE actually copying his books, in order to train on them, and they are arguing that this is okay because they don't want to have to obtain a legal copy of the work, because that would be too expensive.

They are literally arguing they should be allowed to illegally download stuff so long as no human reads the downloaded work directly.

To further your analogy, it would be like your prospective writer objecting to being arrested for stealing the books from the bookstore because he just wanted to become a better writer by studying them, not read them for entertainment purposes.

Re:OpenAI's justification for why training data is fair use,... by pests on Thursday October 05, 2023 @02:09PM (#37781947)
This reminds me of the Google Books lawsuit back in the 2010s. Google didn't ask permission from every copyright holder but was still allowed to scan books due to fair use.
Re:OpenAI's justification for why training data is fair use,... by ang_cire on Thursday October 05, 2023 @02:39PM (#37782344)
The big difference is that Google legally obtained the copies of the books they scanned. OpenAI did not. Publicly-accessible != legal to download/ copy.

The copyright infringement in Google's case was not illegally obtaining the original work, it was about the snippets they provided.

In OpenAI's case, it is also about the illegal obtainment (downloading) of the original work, which they admit to in their brief, but argue shouldn't be illegal if they intend only to transform it. That element was not present in AGI v Google.

Re:OpenAI's justification for why training data is fair use,... by kelseyfrog on Thursday October 05, 2023 @02:31PM (#37782238)
This very lawsuit is referenced in OpenAI's memo as precedent

> Perhaps the most compelling case on point is Authors Guild v. Google. There, search engine Google digitally scanned “tens of millions of books” without their authors’ permission for inclusion in a searchable digital database.

Re:OpenAI's justification for why training data is fair use,... by troupe on Thursday October 05, 2023 @02:49PM (#37782482)
If you are saying that they illegally obtained King's book, then the issue is the fact that the stole the book from somewhere. I don't disagree with your assessment there. But if they checked the book out from the library and let the LLM read it, the fact that it has read it shouldn't be any different than if you had read it as a human.

If I break into the court house to let my child see the artwork in the public areas, the issue isn't whether my child can see, learn from, and copy the style of the artwork. The issue is that I broke into the court house instead of going in the front door when it was open to the public.

Re:OpenAI's justification for why training data is fair use,... by ang_cire on Thursday October 05, 2023 @03:17PM (#37782972)
> if they checked the book out from the library and let the LLM read it

Did you know it's actually illegal to photocopy books you check out from the library? It's an illegal copy.

That aside, the analogies between humans and ML models fall apart when you actually get into the details, which is why AI companies don't want that.

Human brains don't have RAM and Disk storage that keep direct copies of works. Computers do. Human brains are not entirely deterministic calculators. Computers are.

If you have all the state data of an AI model, you can know exactly what it will produce, and the input data is a necessary component of that state data, so it's not transformative in the same way as a human brain in the first place.

2 humans will not create the exact same output given a shared starting state in the way that 2 computers (AI models included) will.

The only reason people falsely believe AI models are different is because no one actually traces through that state data, and there are enough pseudo-random variables involved and state-changes that they create the illusion of non-determinism, but that is, in point of fact, an illusion.

Re:OpenAI's justification for why training data is fair use,... by troupe on Thursday October 05, 2023 @03:38PM (#37783273)
You are assuming that an LLM is saving a copy of the book it reads.
Re:OpenAI's justification for why training data is fair use,... by pclmulqdq on Thursday October 05, 2023 @05:24PM (#37784477)
LLMs don't read, they are trained. Yes, training involves saving a copy of the training corpus.
Re:OpenAI's justification for why training data is fair use,... by denton-scratch on Thursday October 05, 2023 @05:59PM (#37784811)
The LLM is the output of the training system; the LLM doesn't do the reading, the trainer does. The LLM doesn't "contain" a copy of the book.

But some LLMs seem to be able to quote on demand from a wide range of literature; these models evidently do have some kind of memory, even if it's been distilled and blended and distilled again.

Re:OpenAI's justification for why training data is fair use,... by ang_cire on Friday October 06, 2023 @04:33PM (#37796014)
No, I am actually reading OpenAI's PDF linked here, where they say they make a copy of the online corpa to train on, right at the top of page 2:

> By analyzing large corpora (which necessarily involves first making copies of the data to be analyzed

Whether they retain that later is immaterial to whether copyright infringement occurred. Deleting a movie you illegally downloaded after you watch (train on) it doesn't make it legal.

Re:OpenAI's justification for why training data is fair use,... by troupe on Saturday October 07, 2023 @03:22PM (#37804682)
I agree with you. What I'm trying to differentiate is between the way you obtain a book and whether training an LLM on a book that you own the rights to read should be considered copyright infringement.
Re:OpenAI's justification for why training data is fair use,... by ang_cire on Monday October 09, 2023 @02:20AM (#37817460)
Unless the license you have for the book includes rights for commercial usage, it should be illegal to train a model that is being used for business, imo. In terms of individuals training models for their own personal use, I think that becomes much more tough to answer, but that's not really cropping up in the legal fights.
Re:OpenAI's justification for why training data is fair use,... by pests on Thursday October 05, 2023 @02:08PM (#37781929)
> Studying an author's corpus in order to be able to produce works like the author is not copyright infringement.

You didn't finish the analogy.

Not just produce "works like the author" but being able to regenerate those original works word-for-word.

Is that copyright infringement? If I have the entire Dark Tower series memorized word-for-word, and reproduce passages or entire chapters on-demand for others. It's just the way some people learn.

Re:OpenAI's justification for why training data is fair use,... by troupe on Thursday October 05, 2023 @05:53PM (#37784756)
Well yes. If you reproduce passages from a book (from memory or otherwise) and present it as your own, that is copyright infringement. We have lots of well established law about how much things can be similar without infringing.

LLMs that are just spitting out full copies of what they were trained on aren't functioning correctly or haven't been trained on enough content.

Re:OpenAI's justification for why training data is fair use,... by pests on Thursday October 05, 2023 @07:54PM (#37785713)
> LLMs that are just spitting out full copies of what they were trained on aren't functioning correctly or haven't been trained on enough content.

So all of them? None of the LLM's are "functioning correctly"? Because I can get every LLM to reproduce passages from a book.

Re:OpenAI's justification for why training data is fair use,... by denton-scratch on Thursday October 05, 2023 @05:52PM (#37784745)
Yeah, I said that.

You're not the only commenter that has pointed out to me that the model doesn't contain the training corpus; that the model is analogous to a human writer who learns to write by becoming a voracious reader.

I acknowledge that, I think the analogy is fair. The problem is that the memory capacity of the model appears to be humungous; I've never played with ChatGPT, but I understand that it can quote accurately and at some length from a wide range of literature.

By the model, I mean the model behind ChatGPT. I suspect this "memory" amounts to having unimaginable numbers of parameters, i.e. something like overfit. I realize that's just one LLM, and there are other models that are neither generative nor language models, and that have to be much more restrained in what they consider a "parameter".