News for hackers. Stuff that matters.
Re:Collapse of Self-Trained Language Models by Lerc on Wednesday April 17, 2024 @07:43PM (#40071326)
So is there any selection criteria going on here?

It seems obvious to me that training on the totality of the output of a model with any degree of error would result in a greater error due to the error in the source compounding with error in the training. That is simply the photocopy of a photocopy problem.

If there is a selection mechanism then you are reducing the data set to the best of it's output which by definition has the potential to be better than the original training data because divergences from correct prediction of the training data (errors) have a probability to be better than the original training data.

This is analogous to evolution where most errors are bad and some are beneficial, natural selection provides a mechanism to pick the good bits.

Re:Collapse of Self-Trained Language Models by jgalt212 on Wednesday April 17, 2024 @08:34PM (#40071712)
> So is there any selection criteria going on here?

I think that's Mistral's secret sauce.

> They appear particularly adept at marshalling training data—the second ingredient of AI success. Mr Mensch will not say how exactly Mistral curates its training sets;

https://www.economist.com/business/2024/02/26/meet-the-frenc...