In Github's case for example, Github's TOS already includes a clause that if you upload code there, you grant Github a license to use the content to run Github's services... and copilot is one of them.

https://docs.github.com/en/site-policy/github-terms/github-t...

Such clauses are commonly found in social media where users can upload content. Think of imgur, instagam, reddit, etc. OpenAI might buy reddit, and declare ChatGPT a product of the reddit service, then all discussions on reddit could be used for the training of ChatGPT... while open models can't access the data.

You can't grant permission for something you don't own. Uploading a copy of a GPLed work to GitHub does not grant GitHub permission to ignore the GPL. (It might grant GitHub permission to ignore your copyrights in that work, maybe, though it seems like a stretch to argue that "run Github's services" includes "give other people derivative works of all your code"; arguably that ought to be too broad for a contract of adhesion. There's case law about what you can and can't do in a unilaterally imposed contract such as a ToS; a ToS can't say "you owe us $100 if you browse more than twelve pages" either, and codebases can be worth far more than that.)

If GitHub started saying "one of our services is to give people access to copies of your code with the licenses and copyright notices removed", the GitHub ToS wouldn't suddenly make it acceptable to run that "service".

But I agree with part of your underlying point. All AI models should respect Open Source licenses. It's a problem if some try to work around that.

Such a strict interpretation of copyright would kill Open Source, as any attempt at reverse engineering would be disallowed by that.

Furthermore it would mean that only big and rich companies would have AI, as they can do behind closest doors whatever they want with AI, Open Source licenses only cover redistribution after all, which internal use doesn't fall under. Meanwhile any attempt at publicly available open AI models would instantly get killed by copyright claims.

Not at all, for multiple reasons. Reverse-engineering already has that problem: if any of the people who do the reverse engineering also work on the code, it's entirely possible for some of the reverse-engineered code to end up in the new code, making the new code a derivative work of the original. There are standard ways to carefully avoid that: https://en.wikipedia.org/wiki/Clean_room_design

(Also, reverse engineering is a small fraction of Open Source.)

> Furthermore it would mean that only big and rich companies would have AI, as they can do behind closest doors whatever they want with AI, Open Source licenses only cover redistribution after all, which internal use doesn't fall under.

The moment they distribute anything written by the model, the same problem applies. And if they don't redistribute anything written by the model, then sure, they can do anything they like, just as you're free to internally combine GPLed and proprietary code if you never ship the result. (Note, though, that many companies have figured out it's a bad idea to do this, because it creates a combination you cannot ever distribute, and circumstances might change in the future to lead you to want to distribute it.)

> Meanwhile any attempt at publicly available open AI models would instantly get killed by copyright claims.

No, they just need to actually pay attention to the licenses of work they train on. Train on permissively licensed code, document every codebase trained on, and record the licenses and copyright notices.

How is AI "reading" code different from me reading code? Is the difference the AI's ability for perfect memory?

I can read open source code, (even GPL) and not have all future code I independently write be subject to that license. I don't think anyone would argue that I immediately "forget" any OSS code that I read, so it's becoming part of the structure of my brain (and potentially influencing future code I wrote), but unless I'm linking to the code or copying pieces out, verbatim, I'm generally in the clear. Of course there are some sticky situations clean-room, reverse engineering, but those seem like pretty narrow examples.

(I’m not really interested in arguing whether that’s all they do, or whether it’s the purpose of LLMs—those details are just a distraction from the original question: what makes LLM training different than a human reading code.)

If the model has memorized the training set and can reproduce it verbatim when prompted, then it should be incumbent on the AI owner to prove that it does not reproduce copyrighted code when it is not explicitly prompted.

I think the most likely outcome will be to treat AI just like people. They're allowed to learn from any code they can see, but that doesn't mean that if they reproduce a copy from memory that it is somehow free of its original copyright.

That's very consistent with how copyright law already works.

This will leave AI users in a sightly awkward position where they are responsible for figuring out if they unknowingly used AI to unknowingly copy code, but it's not like that can't happen already - as soon as you hire a programmer you might be unknowingly allowing copied code into your product.

If we just zipped up the entire training data set and distributed it with the model then it would clearly be a copy and/or derivative work. The LLM does the same thing as zipping (i.e., compresses the training data…by encoding it in the model weights). Folks just seem to think that it’s not a derivative work because an LLM _also_ does more than that sometimes (e.g., extrapolates from the training data to produce novel token sequences as output).

Why not? Humans store information in their brains that they have learnt. So do AIs. What exactly is the difference between a weight in an Artificial Neural Network and a weight in a Natural Neural Network?

If the answer is "humans get special treatment" then that's fine I guess but I think it's worth being explicit that that's the difference.

> The LLM does the same thing as zipping (i.e., compresses the training data…by encoding it in the model weights).

It's not at all the same. It's highly lossy. Only extremely highly repeated works get memorised exactly and even then it's often not exact.

LLMs do not contain a copy of all the training data (if trained properly). I agree if that was the case then it would be different, but that isn't how they work (unless you badly overfit).

That's absolutely the difference. Humans aren't copyrightable; the alternative would be unconscionable.

> even then it's often not exact

You don't have to copy something exactly to be a derivative work. "Lord of the Rings but a random 15% of words are replaced with gibberish" is still a derivative work of Lord of the Rings. So is "Lord of the Rings but every word/sentence is paraphrased".

An LLM on the other hand generally works off of a model that was trained first, and that model can be saved to a file and read out later. As a result, it's a derivative work that we can examine, copy, share, modify and do all the things with that we generally attribute to something being a Work. The question on if binary output from a program can be copyrighted is somewhat unclear, but from what I've heard legally (not legal advice, I Am Not A Lawyer), it seems to be the case unless you explicitly say it's not[0].

There's a few other things to consider like how you, as a human, can make the conscious decision to avoid specifically replicating GPL code that you've seen if you're not allowed to use it (whether that is by restructuring the code, doing the same techniques in a different language, or the heaviest example which is clean-rooming it). AIs don't have the ability to make that distinction (and to my understanding due to how they work, the only way you can meaningfully avoid it is if you ensure that the entire model is compliant to avoid the AI going off on it's own tangent and making the decision to include incompatible code.)

From a more practical perspective - Copilot will happily spit out and apply the wrong license to Quake IIIs fast inverse square root algorithm function. It's GPL licensed code but it IIRC claimed it was BSD licensed? That alone would constitute a violation and it'd be weird to not point at the people who trained the model that allowed it to make that choice.

To be fair, right now a lot of this is up in the air and all we have to go on is kinda wishy-washy guidance from copyright offices (which is mostly just refusing registration on the basis that a copyrighted material has to be made by a human, not by a machine). There's a couple of ongoing lawsuits specifically about Copilot that are still pending and from what I last heard, the judges aren't very impressed by the defense of GitHub/MSFT/OpenAI. The approach also greatly differs per country/governing body - Japans government has for example given blanket permission for non-commercial AI training, while keeping a strict eye on anyone trying to use it for paid services, while the EU is passing legislation that seems to mostly lean towards "it's copyrighted, that's now your problem to get in line with it", without outright saying it yet.

[0]: This is the main reason why for FOSS, the Creative Commons License usually is not seen as a good pick outside of assets, because it can interfere with distributing compiled versions of your code.

This is incorrect; it doesn't matter whether it's commercial or non-commercial, and you can use anything as training data regardless of copyright. See the amendment of the copyright law from 2018.

By doing so with the explicit intent of building derivative products from it, and at massive scale.

I think this is an incredibly short sighted take.

Training a modern state-of-art LLM needs terabytes of training data. It's probably not going to be practically possible to actually license this much data. And even if we assume it will be then the only entities which will be able to do it are the world's biggest corporations.

So if AI models are a derivative work of their training data then as a consequence the whole field will mostly die overnight, with perhaps only a few of the world's richest corporations being able to play with this space.

I'm sure you're aware of the slew of really cool open/free models that are out there, which you can download today and play with on your local machine. Like Stable Diffusion. It was trained on all rights reserved data. Now it's going to be illegal. Or OpenLLaMA/Falcon/MPT/etc. Also trained on all rights reserved data. Illegal too.

Have you heard about the Pile dataset[1]? It's the most popular open dataset for training LLMs, and essentially every non-proprietary LLM is trained on it, or on parts of it. Do you know that it contains 100GB of all-rights-reserved pirated ebooks? If AI models are a derivative work of their training data then all of those models are now illegal.

[1] - https://pile.eleuther.ai

This would completely kill any chance of having good free/open models. Sure, you could then grab all of GPL'd code, train a model on that, and maybe have a decent working GPL'd model that can emit GPL'd code. There might be enough data for that. But that's it. What about other kinds of models? Image generation, chat bots, personal assistants, story writers, etc. There's just not enough freely licensed data (and probably will never be) to train those.

We must democratize this space. It's already insanely expensive to train state-of-art LLMs; we don't need to make it even more expensive. It's not going to stop OpenAI. It's not going to stop Microsoft. They'll figure it out. What it'll stop is everyone else, and will make this technology completely out of reach for everyone who isn't an insanely rich multinational corporation.

With all due respect, to everyone who's complaining about Microsoft using your code to train Codex, I'll be blunt: you're advocating for collectively shooting all of us in the foot. I don't want to live in a dystopia where only huge corporations will have access to cutting edge AI technology, so please stop pushing in that direction by advocating for more draconian copyright just because you're butthurt that Microsoft/OpenAI used your code as training data without asking for permission.

Big corporations are not the only people doing this; the little guys (e.g. EleutherAI) who release free and open source models do it too. If you try to block the big bad evil corporation from doing it you'll also block the little guys. The major difference here is that the little guys will get completely screwed by this while the corporation has the cash to try to get around it.

I think if anything, this helps to demonstrate the absurdity of ancient copyright law in our modern world. I would rather move in the direction of abolishing copyright rather than in the direction of strengthening it.

As do I, but this is not expanding copyright. This is saying that as long as copyright exists, AI model training has to respect it too, and can't be a laundering operation for license violations. Otherwise, you're giving companies building AI extra permission to ignore Open Source licenses and small copyright holders (e.g. independent artists, authors of stories/comments/text on the Internet), which is an asymmetry in favor of companies with the resources to train huge models.

I'm all for abolishing copyright, and I think it is absurd. What I'm against here is the asymmetry of keeping all the harms of copyright around while letting AI training violate it.

From my experience this is incorrect; it's not an asymmetry in favor of companies; if anything it's an asymmetry in favor of small players.

How do I know? Because I myself am one of those small players, and because I train machine learning models myself. Not huge ones of course, but ones which need data in the range of e.g. 10~25GB range. As an individual it would be completely impossible to explicitly license this data for training (most of which is owned, mind you, by huge corporations!), while a corporation wouldn't have a problem with it.

> What I'm against here is the asymmetry of keeping all the harms of copyright around while letting AI training violate it.

This is fair enough. But please beware that the blast radius of entities your proposal would harm is not limited to huge corporations. I would prefer to fundamentally reduce the scope of copyright too instead of carving out special cases for AI training, but we both know that isn't going to happen.

I'm well aware that this is widely done by large and small entities alike; I'm not just concerned about the practices of large companies, I'm also concerned about small ones, and individuals.

This is not "more draconian" copyright; this is not a change to copyright at all. This is the same copyright we already have, equally enforced for all copyright holders, large and small. You want to "democratize" this space? Get rid of copyright, and a lot of things become better and easier, not just AI.

I don't want the dystopia where copyright still exists for large publishers and studios and proprietary software companies to prevent sharing and remixing things, but at the same time all the small and Open Source entities don't get to set their own terms because AI will just remix them away.

Do you think you'd get away with training an AI on a pile of Disney and Pixar movies? Or is it just the myriad of smaller copyright holders, like independent artists, that you're comfortable stepping on?

> It's not going to stop OpenAI. It's not going to stop Microsoft. They'll figure it out.

Will they? If there isn't, in fact, a legal solution, they're not in any better shape than anyone else. If anything, they're in a worse position, because they have deep pockets for potential lawsuits, while non-commercial efforts tend to not be interesting targets to sue (at most, they get shut down, and others pop up in their place).

Question your assumptions about the world that results from requiring AI to respect Open Source licensing and other small copyright holders such as independent artists or online comment/story authors. It's not a corporate dystopia. It's a level playing field.

Maybe it's not evident to non-practitioners in the field, but to every serious practitioner it is obvious that you can't train a state-of-art model without a lot of data (at least right now without some colossal breakthrough), and that actually licensing that data is not really practical (because you need terabytes of it), and it's definitely going to be impossible for anyone who isn't a megacorporation.

Can we agree on this point? If not can you please explain how do you think that e.g. a single individual like me will be able to train e.g. an image diffusion model (so I'd need a few terabytes of images) if I have to respect the licenses of every image in the training set?

Okay, so I hope we can agree that it won't be possible? So now here's the question: do we want such AI models to exist, or do we want to make them illegal (and maybe available only to huge megacorporations)? These are our only two choices, which logically follow from the requirement that we need a lot of data for training.

What I'm advocating for is that we should allow such models and that they're beneficial to us as a society, hence the "and we must be able to do this" in my argument.

I'm starting with the assumption that I want these models to exists and that everyone should have access to them, and then go backwards from that. What you're starting with is the assumption that the training data copyright should be respected, and you're going backwards from that. But these two graphs are not connected, which is why we can't agree.

Or in other words, what you're (indirectly) advocating for is to make those large models effectively illegal. This is, of course, a valid stance, and if you want to take it then you're free to do so. But that's objectively what you're proposing in practice, and personally I disagree with it.

> Do you think you'd get away with training an AI on a bunch of animated Disney movies, and asking it to generate new images in that style, and using the result in commercial endeavors?

Yes.

Just the same as if I'd draw an image in the style of an animated Disney movie by hand.

In both cases I'll be sued for trademark infringement if the image's of the Mickey Mouse though.

In many cases the current "inequality" of how law is applied to individuals and to megacorporations has little to do with the law itself, and everything to do with how rich the megacorporation is. Try to set up an apple orchard and pick an apple as a logo[1] and tell me how it goes. The law explicitly states that another company, say one which produces computers instead of actual apples, has no merit here, but alas they have deep pockets, so here we are.

[1]: https://www.wired.co.uk/article/apple-vs-apples-trademark-ba...

> Question your assumptions about the world that results from requiring AI to respect Open Source licensing and other small copyright holders such as independent artists or online comment/story authors. It's not a corporate dystopia. It's a level playing field.

Well, let's see, for the sake of argument let's assume that the current widely believed legal status quo is true. (That is, that you can train a model on any data regardless of copyright because it's fair use. Although in my country that's explicitly allowed by law so here we don't have to assume anything.) Right now OpenAI can scrape 1TB of data off the Internet and legally train a model. I can also scrape 1TB of data off the Internet and legally train a model. And it can be any data, not just open source programs and content produced by small copyright holders. Is this not a level playing field?

Are you seriously suggesting that having to pay billions of dollars to license the training data necessary to train a model is a level playing field? I guess if nobody will be able to do it then it will be, in a way, a level playing field; I just fear that entities with enough money will be able to license enough data anyway and then the rest of us will end up with nothing.

Yes. Creating a new animation that is inspired by Disney's style is clearly fair use. Otherwise Studio Ghibli would be in trouble.