Yes, and if we reproduce something significant that way we have (depending on licences etc.) committed copyright infringement. The definition of significant is a very grey hairy one though – there are many obvious examples where you would say it is definitely copying from memory (so plagiarism rather than research or accidental similarity) or copying directly, and many examples where we'd all agree that the result is so likely to be independently produced that it isn't an issue, but even more examples where there is room for interpretation and disagreement.

This isn't as well-defined for humans as you might think, so can't be well-defined for LLM techniques by comparing them to human agents.

My argument against the current hoovering up of data under various licences for AI training, which they claim can#'t reproduce anything verbatim, is CoPilot. If there is no risk, then why did they only use public repositories and none of their own private ones? Surely they think their code contains good training material, unless they think their own code is gobbledygook. Or back in terms of licences: if it can't breach the GPL family, then it can't breach their own commercial licensing arrangements.

Would OpenAI have a problem with humans for using some of their code/documents/other in this way?