The mistake you make here is to forget that the training data of the original models was also _full_ or errors and biases — and yet they still produced coherent and useful output. LLM training seems to be incredibly resilient to noise in the training set.

Nonsense. Some of the current best AI models were specifically trained on AI output.

Forget what it eats to continue improving.

Realize what it already has.

A foundational language model with no additional training is already quite powerful.

And that genie isn't going back into the bottle.