I agree. Basically, the issue is not that AI can reproduce _ideas_ but that it can execute ideas in the same way as the original, or with parts of the original. It doesn't learn. If it could learn openai would create a new programming language, train it for real, and it would be the greatest programmer. The marketing that is it is "intelligent" and a "black box" that does things on its own is just science fiction. A hallucination on the part of those promoting that type of ai.

I'm not sure you can ground a copyright judgment based on whether something is truly intelligent, or just looks like it.

The means through which a piece of content has been produced is totally irrelevant. The original content has to be distinguishable in the new one for copyright laws to apply. This has been judged over and over when sampling became more and more widespread. Some DJ sampled the original songs blatantly and thus had to pay the original authors. Others applied so many sound transformations that it was basically impossible to identify and thus copyright was basically impossible to apply.

This is not true, it clearly does learn

“If it could learn openai would create a new programming language, train it for real, and it would be the greatest programmer.” I don’t understand what you’re saying here. It can invent new languages, but they would those be better than existing ones? It’s less intelligent than a human (though still intelligent) so the language would be worse than existing ones