At what point you can't copyright an "AI brain" either? Maybe AI will at some point create works subject to copyright?

What's the difference between using a pencil to write something and using an LLM to write something? Seriously, I'm asking the question. Why does one produce something copyrighted why the other doesn't?

> What's the difference between using a pencil to write something and using an LLM to write something?

The pencil is not a derivative work of a pile of copyrighted material.

> Why does one produce something copyrighted why the other doesn't?

There's existing case law that non-human entities (e.g. animals) can't create copyrightable works. And in the case of an AI LLM, the AI LLM itself is a derivative work of its training data (as evidenced by the fact that it can by default spit out training data verbatim, even if it has had after-the-fact filters added to prevent such responses).

The copyright office has issued guidance on this which contains a very thorough and thoughtful legal analysis; you would probably be most interested section 3: https://copyright.gov/ai/ai_policy_guidance.pdf

The practical answer is that the copyright office refuses to register AI generated works, and you can't sue for copyright infringement without valid registration under Title 17.