For instance, a recommendation system's output effects what users see, so it effects what they click on. The next training set is statistically dependent on the input of the previous.
There are strategies to deal with it, but in the case of a LLM it seems difficult apart from downweighing everything after 2023 that isn't from a vetted source.
https://en.m.wikipedia.org/wiki/Low-background_steel
tl;dr steel (except for low background steel from sunk battleships) has been pretty much forever contaminated by nuclear weapons.
I'd posit that data-wise, ChatGPT is the equivalent event.
The RLHF reward model is effectively a good/bad content filter. Human==good AI==bad doesn't always hold, there is good AI generated content and bad human written content out there, we should filter by content quality and novelty, not origin.
If I had the resources I would turn GPT-4 onto its own dataset to mine all the facts from all the source documents. Then flip the index grouping by fact, and for each fact it should write a page listing the supporting and contradicting evidence. This would be super useful in determining - 1. existence of a fact, 2. its controversiality level and 3. the distribution of answers.
With these pieces of information at hand GPT would get much better, it would at least stop hallucinating. This fact index would probably double the size of the training corpus as well. Google, with its years of search logs, is the best positioned to create this fact index. They already had simpler attempts at creating a knowledge graph for a long time. But if they don't move to exploit the data they sit on, others will take the lead anyway. It costs only money to run so much text over the model, no slow & expensive human involvement necessary.
It's almost certainly a problem for LLM development, just like it is for humans. Humans generate all sorts of stuff, some of it being absolute bullshit, and it does seem to cause problems for other humans. Yet with effort it still seems to be possible to cut through the bullshit in a lot of cases, so it's likely not an insurmountable problem for LLM development either.
We have to be very careful with the "factual" content of old non-fiction works. So much of that, from history, to medicine, to biology, etc, turned out to be straight from their writers' imaginations. If an LLM considers such books to be no different from modern books on these subjects it would get a very skewed view of the world.
Imagine asking an LLM for a medical diagnosis and it responding with something about the humors.
For this, large amounts of very old text would be extremely valuable.
What about data that's sitting around and isn't supposed to be public? If training data gets scarce, does a market for small-medium sized data emerge? Like old homework papers, internal company documents, etc?