In other words you can never be sure if synthetic data is any good or if what things gravitate toward are really most optimal.

I wouldn't ever make "most optimal" a criteria. We're looking for measurable improvements, not a jump to god emperor or apex predator.

> you may only be in a localized threshold of fitness stability that is not necessarily optimal, but separated from another optimal configuration by having suboptimal intermediary steps that need more activation energy to overcome before falling into a state with lower entropy (or more optimal fitness).

Optimization is like that. But unlike genetics, where we can't re-route the recurrent laryngeal nerve or change fundamental biochemistry, these are engineered systems where we can set up wildly different experiments at any time. Just to cite one of many different research threads, there's now research now going into developing models from small scale training data.

> you can never be sure if synthetic data is any good or if what things gravitate toward are really most optimal.

We can know if the synthetic data is better. We have objective measures, a scientific process, and we'll always be striving for improvement.

You don't even need to go that far. How do most children learn? By reading textbooks and listening to lesson plans assembled by their teachers from all the relevant content the teachers have experienced.

Our education systems are built on synthetic data that is created for optimized learning, so that every child doesn't have to prove the universe from scratch to learn some basic maths.

not sure how effective that would be, if it was his only source of learning.

The SOTA is to use a discriminator (often another LLM or ML algo) to select the best output before feeding it into the training data. That’s what OpenAI, Anthropic, et al have been doing. One of them just published a paper about it a few weeks ago.

You could describe a textbook as a synthesis, sure, in a sense which absolutely does not track with the 'synthetic' in 'synthetic data'.

Unless the textbook is AI-generated, and I expect that in 2024, the number of AI-generated textbooks is not zero.

But before humans can understand enough language to ingest that synthetic data, they do a lot of their own discovery based training where they learn about the world physically and absorb the language people around them use, kind of like throwing random internet data at an LLM.

However, I do get the spirit of the article, that as more information generated online is done by LLms, the validity and use of the output decreases

It's the same as any other kind of signal processing. You can increase the noise, but you can't get more signal than you started with.

Here, if the LLM decides that "monkey" is most often followed by "butt" and occasionally by "trainer", then it'll generate synthetic data with those frequencies and training on that data will not change its probability estimates at all. It will, however, drown out the signal that "you are a monkey butt" is more likely than "phlegm cigar monkey butt", if you'll forgive me the liberty of using those phrases to represent statistical correlations just beyond the frontier of what the LLM has learned. The synthetic data will teach it that everything it doesn't already know is equally probable, which will overwhelm human source data in which it isn't.