News for hackers. Stuff that matters.
Re:OpenAI's justification for why training data is fair use,... by rpdillon on Thursday October 05, 2023 @01:26PM (#37781372)
The models that are trained on these inputs absolutely do not contain all of the text that they're trained on. Even with non-quantized models you're looking at somewhere in the realm of 3% of the training data being present in the final model.

Further, these models are not used to reproduce copyrighted works... they would be one of the least efficient ways to do so.

I also don't think there's a particularly strong argument that training on a book or image decreases the market for the original book or image. To take an example, zero folks that want to read Game of Thrones are asking ChatGPT to generate Game of Thrones for them instead of buying the book.

I think what's really happened here is that artists' work has utility outside of the originally intended audience, that utility largely comes from statistically analyzing the writing, and the debate is about whether or not the utility of those statistics requires additional compensation for the artists. That question seems to hinge on the intricacies of copyright law, but as I've stated before, I think the fair use argument is quite strong.

Re:OpenAI's justification for why training data is fair use,... by pbhjpbhj on Thursday October 05, 2023 @02:08PM (#37781932)
>I also don't think ... that training on a book or image decreases the market for the original book or image. ... zero folks that want to read Game of Thrones are asking ChatGPT to generate Game of Thrones ...

For now. But when you can generate work in the style of an artist without paying that artist, I would. More importantly, media production companies will and so will use access to people's post work to put them out of work now.

We need to distribute the benefits, we're drifting towards copyright infringement being allowed by rich corporation (with resources to generate "AI") as long as they abstract it through a transformer, or whatever.

Re:OpenAI's justification for why training data is fair use,... by antifa on Friday October 06, 2023 @09:04AM (#37790239)
Crypto was a decentralized ponzi scheme. AI will be decentralized mass copyright infringement.
Re:OpenAI's justification for why training data is fair use,... by artninja1988 on Friday October 06, 2023 @09:56AM (#37790859)
Copyright laws are fascist and should be abolished
Re:OpenAI's justification for why training data is fair use,... by Turing_Machine on Friday October 06, 2023 @09:48AM (#37790759)
> But when you can generate work in the style of an artist without paying that artist, I would.

You can't copyright a "style".

Re:OpenAI's justification for why training data is fair use,... by joaogui1 on Thursday October 05, 2023 @02:54PM (#37782576)
Where does this 3% figure come from?
Re:OpenAI's justification for why training data is fair use,... by Zuiii on Thursday October 05, 2023 @05:00PM (#37784239)
These models are generally trained on tarabytes of data, but are usually 10s of gigabytes large (or much less if quantized). The latest true open source model, mistral 7b, is only 3GB (0.3% of a TB) when quantized.
Re:OpenAI's justification for why training data is fair use,... by rpdillon on Thursday October 05, 2023 @07:28PM (#37785527)
I did a very similar analysis with Llama 65B being trained on 5.6T tokens assuming token length of 4 characters and comparing with a quantized model size of ~38GB.

The 3% number was a conservative rounding of the same calculation, but retaining fp16 rather than quantizing to 4 bits.

Here's my original back of the napkin analysis:

https://news.ycombinator.com/item?id=36681440

Re:OpenAI's justification for why training data is fair use,... by pclmulqdq on Thursday October 05, 2023 @05:21PM (#37784449)
> To take an example, zero folks that want to read Game of Thrones are asking ChatGPT to generate Game of Thrones for them instead of buying the book.

Why jump straight to one of the longest popular books in English? I'm sure plenty of people who would otherwise buy poetry anthologies are asking ChatGPT to write them a haiku. It's also entirely conceivable that ChatGPT can perfectly reproduce a poem in its training set - it can produce paragraph-sized quotes.

Re:OpenAI's justification for why training data is fair use,... by rpdillon on Thursday October 05, 2023 @07:30PM (#37785542)
It was a reference to the copyright suit brought by the author.

https://apnews.com/article/openai-lawsuit-authors-grisham-ge...