Yes although how close to verbatim is debatable. For example there are questions that you’d ask that other people have asked many times before that you’d like the exact answer for (e.g. when does daylight saving time end?)

> that make it forget specific facts. These are often meant to satisfy copyright claims

Facts are not copyrightable.

To quote copyright.gov: “Copyright does not protect facts, ideas, systems, or methods of operation, although it may protect the way these things are expressed.”

What is a fact without expression? It's not clear under what interpretation might be necessary to get the quoted sentiment to be considered sensical.

Wouldn't that be stuffed in the prompt anyways? No reason for the LLM to learn that.

It really depends on which part of the corpus, though. I do expect my LM to be able to reproduce culturally important citations, for example.