> These language models aren't actually smart, they're just good at parroting smart sounding things they've been taught (although one could argue humans often do the same thing); the problem here is they need more sources to intersplice to hide the true origins.

That's debatable, and it's not necessarily true. Just the other day, we saw a story here (https://news.ycombinator.com/item?id=34474043) that suggested that a language model trained on Othello game transcripts did in fact build a latent-space "mental model" of the board state. By perturbating the latent space, the researchers could argue that the model wasn't simply parroting memorized game sequences.

Of course, this isn't to say that AI plagiarism is impossible or even unlikely. Plagiarism is a shortcut to acceptable quality output, and AI systems excel at finding shortcuts to acceptable output.