This never made sense to me. Seems analogus to inbreeding in biological organisms.
More complex synthetic data generates things that could be in the training data but aren't for some reason. For example, machine translation training corpuses often contain a lot of formal language (because formal documents are the ones that get translated), so you might generate a bunch of casual conversation to improve the translation model's ability to handle that.
Wasn't model collapse/model decay a big issue since the beginning? i know that RL kinda mitigate the effect, but if this was close to not be an issue anymore, wouldn't we have a paper on that already? All that i've seen on the subject was about mitigation, never about solving the issue.
Sorry if my question is dumb, I need to do some research about the difference between image generator and text generator, i feel like my understanding of LLM is skewed by my usage and understanding of Z-turbo, but i only have that much time in my week.
The hope was that weak recursion could produce strong recursion by automating the iteration of ML pipelines.
Theres also the idea of LLMs producing synthetic data to contribute to new models. Perhaps that is not as effective as hoped.