Agreed. The irony is that all the current anti-machine-learning lawsuits will make it so only big companies with deep pockets will be able to continue to make & use them.
My take on the matter is that facts aren't copyrightable. A LLM is basically recording facts.
I have a background in stylometry where you basically determine authorship by recording statistics on known pieces of writing to try to find the author of another piece of writing. So just extracting facts. You could easily now generate random sentences and filter ones that pass a certain threshold. LLMs are basically this on steroids. So it's just facts and data, not copyright infringement.
Otherwise, you'd have to argue that software reading the text contents of a book are copyright infringement.
The logical jumping happens around this period symbol is spectacular.
You measure things like:
* What average sentence length?
* What is the ratio of adjectives to nouns?
* What percentage of sentences are in a passive voice?
* What is the distribution of words used?
* Is "the" used more often by the other than in general usage?
* "whence" vs "when".
* "tyre" vs "tire". Etc.
This could all be printed out and sold as a book and each measure would be a very boring fact that is not copyrightable and contains no copyrightable content.
I could give you a word frequency list from the A Song of Fire and Ice series and George RR Martin could do nothing about it. (Technically, he might have the rights to "Arya" and "Lannister", but to say "the word 'Lannister' appears 1,337 times in the series" would clearly fall under fair-use).
1. Anything existing is considered a fact.
2. Stating a fact doesn't infringe copyright.
The only conclusion your logic can infer to is copyright doesn't exist.
For example, I can say "the first sentence of A Song of Fire and Ice series is <insert the first sentence>, the second sentence is <insert the second sentence>, the third sentence is...". It's still just a list of facts, so no copyright infringement, right?
But if you went by and grabbed the very first sentence from each chapter that's is sufficiently in the clear. Especially when you are doing something transformative with it. Like for example analysing what makes for an effective opening sentence.
With copyright you always get to ugly slippery slope arguments, Tom Scott has a great video on it.
But if you cannot extract something the replaces the original work you should be already in the clear. I'm pretty sure Cole's Notes don't need the original author's permission to print and they are arguably a substitute for reading the original work. I've skimmed through entire series by reading the individual episode summaries on Wikipedia. These are in the clear.
The fact that a machine speeds up a human process does not change legality.
How so?
How an entity (human or not) changes after reading those books is not in the purview of copyright.
It has not shared them (copied). It has read them, learned from them, and changed itself in response.
Seems like a problem to me.
Man, I wish as a human I could use the defense of "but it was publicly accessible in a torrent" as a valid reason that I acquired and consumed some content.
Nobody is ignoring it, that would be a crime if they did, but since there's no evidence of it... you're being upset at your own hypothetical.
I see articles all over that LLMs for example used books3, which the creator himself has admitted came from torrenting from bibliotik, and which contains at least some books that are not otherwise freely available.
The issue that content creators have with LLMs is not getting proper attribution for their creations and source material that went into training these LLMs, and it seems pretty clear to me that some of the content that was used to train LLMs was not legally obtained and licensed to consume.
In technical terms, these companies are trying to avoid the need to consider content licensing, which is a major violation to content creators.
Content is obviously being relicensed as the models are not fully open.
On top of that, it is also being exposed as pay-for-use subscription models without paying anything to original authors. How is that fair use?
LLMs are trained on tons of books and papers that are not available to humans for free anywhere.
You are telling me that all the books in books2 and books3 can be acquired and read by myself completely free of charge legally without piracy?
If humans can get these books for free legally, why are so many people paying so much to buy them for their college courses?
I thought it's well known that LLMs are being trained on pirated content.
Now if you are saying they obtained the book from illegal sources that is a valid argument. But the issue there is orthogonal feeding the contents of the books into an LLM. It is legal for me to view and photograph the Mona Lisa. If I break into he Louvre to view the Mona Lisa, the issue isn't that I viewed the artwork. The issue is that I broke into the museum.
But if you have an example of some types of books that it is impossible to borrow from anywhere on the planet please share. Textbooks are readily available from libraries.
It's widely known that some of the big LLMs were trained on book datasets that were acquired via torrents. And that those datasets do contain at least some books that are not available anywhere freely via legal channels.
What's illegal is making and distributing copies of a book.
But the reader isn't in trouble for reading a book they don't own.
So I am free to torrent all the books that I want, and there is no legal action that can ever be taken against me?
If so that would be news to me.
No of course not.
Creating copies, and distributing them to other people is against the law.
That has nothing to do with what I said though.
What I said is that it is not illegal for a reader to read something that they havent paid for.
Do you see how that is different from distributing copies of something to other people?
I am asking about reading content that is not available through any free means, and pirating it in order to read it instead of paying for it from the author.
Reading it is legal.
Distributing torrents is not though. That is the part of piracy that is illegal. The illegal part is when you create and distributing copies.
Copyright law has nothing to do with reading stuff that you have not paid for.
Instead, it is about the illegality of creating and distributing copies.
That's why it's called "copyright" law. Because it is about copies.
It is not "readright law".
No, there is nothing illegal about reading something that you havent paid for.
What about movies and tv shows? I can download them via torrents to view without paying as well, completely legally?
Nobody can ever take any legal action against me ever for doing this?
How though would the creator of the content ever get paid for it if everyone can read and view it completely free? Why would they go through all the work of creating it with no gain to themselves?
As far as I know, nobody in this country has ever been successfully prosecuted for the mere act of downloading, or even having obviously-illicitly-acquired pirated media in their possession.
As to your final question, the actual answer is because people are lazy. It requires more effort and technical sophistication to maintain a movie file collection than it does to run Netflix. Nearly any piece of remotely popular content is available for free with a bit of knowledge, but it's less hassle to just subscribe to services. There is no world in which piracy actually endangers creators.
Also, I'd say there is a difference between being illegal and whether someone has been successfully prosecuted or not.
I don't understand though how piracy doesn't endanger creators though. If a small time creator creates something, say a video game, and everyone pirates it instead of buys it, how has that not endangered him? Simply because you don't think it's plausible that enough people would refrain from paying?
They can if you create or distribute copies.
That part is illegal and will get you in trouble, and is why people get in trouble for torrenting.
Just reading or watching other people's content isn't the illegal part.
[0] - https://www.pymnts.com/digital-payments/2023/58percent-of-co...
Copying a webpage into a cache that allows presentation of the page to a user is only allowed because it is part of rendering the page to a user. Even if a computer only copies two words at a time from a source text, if it copies a substantial part overall then it still copied.
Honestly, I don't think slurping data to train ML models is allowed by copyright (but, I do, probably think it should be, as long as any significant reproductions are then prosecuted as infringements [also we should reduce copyright terms to ~7 years!]).
This is all my own opinion, unrelated to my employment.
Almost all modern books are available as ebooks for the Kindle or otherwise. You aren't doing anything to the content itself that could be a violation.
Except it does, and it's in a quite literal sense.
You painted a painting. I took a picture of it and compress it as .jpeg. Byte-wise the .jpeg file has no similarity to your original painting. For someone who doesn't know .jpeg, it's just garbage bytes.
So I didn't infringe your copyright by selling this .jpeg file. The user who decodes the .jpeg file and displays it on a monitor does.
Does it sound right? This is how "weights do not contain training data" argument works.
And before "how about artists who store this information in their brains?" Well human being is a special case for every law in every country. Just like selling a cow's liver is never the same as selling a man's, even they're both organic tissues. A human's brain is always going to be treated differently than a hard drive.
The thing I hate about this whole situation is that it's going to be decided solely on which side has more lobbying power[1], since just as you said, it's something in between.
[1]: One could say all the policies are decided this way...
The silver lining here is that lobbyists can only afford to care about finite number of things at a time - so even if they get some laws their way, other factions can push laws mitigating the damage somewhat.
I currently believe (weakly) that the outcome of more and better lobbying isn't laws getting directly worse for society, but rather the regulatory system grinding to a halt under increasing number of laws that exist to cancel out parts of other laws...
Since portions of such works can be recovered in inference, they contain at least a lossily-compressed copy of the collection of works used in training. Reproduction isn’t pure coincidence.
--
[0] - I believe it's actually fundamentally the same thing.
Yeah, that copies in the brain of a human who has experienced the work and not reduced to any othe media are neither “copies” nor “phonorecords” as covered by US copyright law is pretty clear, and that this does not apply to data stored by clever algorithms in computer storage is also pretty clear, so I’m not sure what your point is.
If you want to argue LLMs are people to escape this, then, sure, copyright stops being a problem for their training data, but it still is for their output plus you end up a whole set of new legal problems with using them as people do now, starting with 13th Amendment problems.
The fact that people making the second part of the first argument often either are ignorant of or deliberately ignore why human learning is treated the way it is doesn't change the essence of the situation.
(Which, knowing history and observing how we treat animals, will happen only when the sentient AIs coerce us to recognize them, through violence or the threat of it...)
“The purpose of law is to advance the collective interests of humans in a society” is, I would argue, a high-level principal, and (with variations as to whether it applies to all humans or some subset, and which interests are considered privileged relative to others) nearly universal. Yes, the fact that humans are writing the laws is a reason that principle is chosen, but the idea that there are not high-level principle is just false.
They would be "legal people" in the sense that corporations are "people".
I'm sorry, this is far from "clear", otherwise we wouldn't be having this discussion. Fair use is a thing. Does fair use apply in this situation? No one knows at this point.
Hear, hear. Unfortunately this is more or less impossible given current copyright law.
Suppose you scrape libgen and turn it into training data, then you release the training data publicly. Since the vast majority of every book appears verbatim in the training data, is this sufficiently transformative?
I think yes, it is, because nobody is going to read those books from the training data. When I made books3, I felt it was important to render each book into high quality text. But it turns out that when you convert Jurassic Park into a text file, there's no good way to read it anymore. Good luck trying to bookmark wherever you left off -- it's all one gigantic file.
But nobody seems to agree. The Danish Rights Alliance (https://rettighedsalliancen.com/) aggressively DMCA'ed anyone that hosted books3, even going so far as to DMCA The Pile from academictorrents: https://academictorrents.com/details/0d366035664fdf51cfbe9f7... with the justification that ~100 copyrighted books appear in the training data, so therefore they have the right to DMCA. Right now most of the world seems to agree, but I'm hoping that opinion will shift as the years tick by. Surely no one can believe that a plain text document poses a serious threat of economic harm to the original author. So the question is whether the original author should be allowed to deny everyone else the right to transform their work into a form that machines can read.
For my part, I've been planning a books4 dataset, but this time similar to LAION: it's a script that spiders libgen torrents (https://libgen.gs/torrents/libgen/) and converts all the epubs into text files. That way, if LAION isn't infringing, then books4 can't be infringing either. (Of course, hosting the actual training data anywhere is pretty hard nowadays, but it should only take a few days to convert 38TB of libgen into ~2TB of plain text.)
This is the only way to create an open source competitor to ChatGPT.
The text file contents all the text in the book in an (at the very least) machine-readable way, it is perfectly feasible to write a program to put it in ebook form or to play it as an audiobook, and then the text file becomes desirable for laypeople to read.
If your (quite reasonable) argument holds water, then it sinks our ability to ever share copyrighted training data with large context windows. That’s why it seemed important to justify a plain text training format, since anything on top of it would be equivalent.
Indeed, one alternate training format would be to ship the raw html from every epub file, then process it into text at runtime. But this makes it trivial to reconstruct the original epub file and use it in an actual book reader.
It’s frustrating that we can’t share the epub files, because there are so many advantages: you can scrape the metadata, you can tweak the rendering to plain text, you can get semantic info from the images (and even OCR them — it turns out that lots of coding epubs show code examples as screenshots, because epub html rendering is so primitive, so this would be the only way to let your model learn from those).
All of that is why I’m leaning towards "make a script to spider all of libgen and cache the epub files locally". But I haven’t finished calculating how much disk space this would require.
I’m sad that researchers will have to wait days for their training data instead of downloading it in a few hours from a high speed cache, but it seems like any such cache would be swiftly DMCA’ed, so there’s no alternative.
The insight is that encoding content is not functionally the same as copying verbatim a work which is the original intent of the concept.
For example: if I have legally obtained a copy of e.g. https://archive.org/details/free_culture, then I am at liberty to encode it in whatever format I need to be able to feed it to a machine/tool for whatever purpose I want. I am not infringing on copyright because I legally obtained the work, and the machine is not because machines can't.
I think at least this much OpenAI has in their favor. If they can prove they legally obtained all the training material (I do think it's fair for them to be required to pay once for it) then I don't think there's any world in which it makes sense to try and allow content creators to further extract royalties from that process alone.
If a user asks an AI for a copyrighted poem, for instance, and the user goes and republishes that poem as their own, I do think it may make sense to grant the original author royalties under current law.
I really hope we can legally pick these two concepts apart and focus on each scenario independently. I see a lot of people here arguing that giving a work that you have legally obtained the rights to view to a machine model is inherently copyright infringement because you had to copy the work and yada yada is it fair use? I really think this is wrong both in the interpretation of copyright and practically as ideas can't be owned and it makes no sense to limit which ideas were used to make what commercial product etc. (we don't do that today and AI doesn't change that).
Only if the training data is under copyright! How about data that is in the public domain, or granted license specifically for training - Mozilla's commonvoice is an example.
Maybe someday there will be a rulings that do for ML training datasets what earlier ones did to legalize cleanroom design.
One way to see this is to imagine a Midjourney competitor trained solely on public domain images. The visual quality of the model will always be worse.
As for licensing, I agree for commercial entities, but there should be exceptions. If a model is open source, it benefits everyone, and so it shouldn’t need to have been licensed. There are a few reasons why this is pretty important, but the main one is that without it, the open source community has no chance whatsoever of creating cutting edge models.
You know, you could probably even argue traditionally that taking all the books in the world and adding them to a corpus for the purpose of creating an LLM would be a transformative work since it doesn't compete with or detract from any of the originals…
There's currently multiple attempts at creating what you describe as books4.
To the extent they are out-competing artists, it's diverting a money flow from artists to somewhere else. But a lot of that money might be from people who would never have bought that art to begin with?
In any case, the money never belonged to the artists. It never got to them. The people it actually belongs to (the customers) get to make the decision about what to buy.
Let assume this is true. It then follows that if we train LLM on all books that are out of copyright, the LLM would be able to replace the "work of the people that produced the content" that is just about to go out of copyright. Feed this body of work back into the LLM and you should end up with something just as advance as what you get by feeding in works that are under copyright.
I'm not sure the assumption is true. But if it is true, using copyrighted material or only non-copyrighted material shouldn't result in a different situation.
Edit: To be more explicit, on my view both the parent and the op are right, and they don't contradict each other: We don't need new copyright legislation. We need new legal concepts.
I think both people are right: There is an overwhelming need to address the situation of artists and writers ("content creators"). Copyright is not being used because it's the right tool to do so, but because it's the only even vaguely plausible one. It's also clear that it's not sufficient.
In what way is that not sufficient?
If creating an LLM is copyright infringement, then we could say that a human reading a book is copyright infringement.
It's obvious that this technology was not considered when developing current legal concepts, so if the concepts produce unsatisfactory outcomes, they need to be reevaluated.
Why do people so easily and readily conflate the two? Is it just because it makes the AI hype sound more real, if you compare it 1:1 with a human brain?
Machines and humans are not equal or even in the ballpark of being similar in any way other than superfluous ones, and should (and already are) held by different standards, both legal and otherwise.
Do you simply disagree that there are issues worth addressing?
Also, typically the entity that carries ideas can not be owned by a cooperation. So there is a completely new relationship between property and ideas at play.
It's clear already that companies are going to be want to have their cake and eat it too. When they leverage AI to produce content, they don't want to be forced to pay licensing fees on the inputs to their models. But on the flip side, they absolutely don't want their content being used in models leveraged by other people.
Not allowing copyrights on wholly or partially AI-generated work is probably the lesser evil path to take.
First of all, even if the models never reproduce any of the copyrighted content verbatim, they will soon be good enough that they will be able to replace the work of the people [1] that produced the content the AI was trained on, like writers and programmers.
Second of all, the nature of the use of the copyrighted content is purely for-profit, and the ability to squeeze the profit out of the best models will stay with the largest corporations -- effectively transferring wealth from the people that produced the content, to the corporations.
We need to find way to make sure the access to this data the models were trained on, and the resulting models, is open and fair. Who wrote which paper [2], or who supplied the initial GPU training, should not really matter that much in the grand scheme of things.
[1] or at least cause downward pressure on their earning potential
[2] you could use similar arguments here they use for using the copyrighted content -- each paper (except maybe the Attention is all you need paper) contributed only marginally