I agree for the most part that 2 is what most people find unacceptable, not 1.

The problem is that, like any general intelligence (e.g. humans), any sufficiently generalized model capable of 1 will also necessarily be capable of 2, regardless of whether it's trained on copyrighted material or not. How do you make an AI model that's capable of summarizing Wikipedia articles but not news articles? Or that's capable of generating consistent images of my original character from a reference photo but not images of Mickey Mouse from the same? This is achievable only by restricting software freedom; by taking measures to prevent users from "running the program as they wish" and from "studying the source code and making changes".

I'll note that the way we have typically enforced restrictions on the behavior of general intelligences in the past (before AI) is to pass laws and enforce punishments if the laws are broken. Not to try to somehow take away people's ability to break the law in the first place, because that would require unacceptably onerous restrictions on human freedom.

I think the same principle applies to AI. Trying to make it impossible for people to use AI to break the law is a lost cause, only achievable by unacceptably onerous restrictions on human freedom. Instead, we should do what we've always done: make certain actions illegal and punish those who do them anyway in violation of the law. Maybe new laws might be required for that in some cases (e.g. deepfake porn) but for the most part I think the laws we already have on the books are sufficient, maybe with minor tweaks.

That all sounds great until you're dealing with deepfakes that come from a country without an extradition treaty?

Not really that different from other forms of illegal content coming from countries without an extradition treaty. (Piracy, scam calls, CP, etc.) Trying to stop it by imposing onerous restrictions on your own citizens isn't likely to be effective.

I would summarize your points as:

We need to create a whole new body of law for enforcing copy write protections in the age of AI.

Does the AI adequately attribute its sources? Does it paraphrase in acceptable ways or just repeat large swathes of text from its corpus with minimal changes?

The laws should force any LLMs not yet capable of complying with these requirements off the Internet until they can comply.

Imagine consultants had to cite sources and pay-out every time they referenced knowledge gained from reading a research paper at working at a formal employer.

I can understand the need to prevent verbatim copying of data. But that is a problem solved on the output side of LLM's, not on the data input for training.

It is completely legal for someone to pay me to summarize the news for them every morning. I can't help but feel that knee-jerk regulation is going to be ultimately bad for everyone.

I think, at one point in time, it was also completely legal to break into computer networks because there were no laws against it.