News for hackers. Stuff that matters.
Re:AI models collapse when trained on recursively generated ... by simonw on Wednesday July 24, 2024 @12:30PM (#41058756)
> We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models

The key word there is "indiscriminate". All of the big AI labs have been training on synthetic data for at least a year at this point, but they're doing so deliberately.

I don't think the "model collapse" problem is particularly important these days. The people training models seem to have that well under control.

Re:AI models collapse when trained on recursively generated ... by mcswell on Wednesday July 24, 2024 @02:13PM (#41059979)
The question (which I raised in a top-level comment before reading your post) is whether there is any such thing as "discriminate" use of web data. Synthetic data created in the same lab as the LLM is discriminate, but what the authors of the paper are saying (if I read it correctly) is that scraping the web is not currently done in a discriminate way. And it's not at all clear to me that there is a discriminate way to use web scraping, because you can't know for sure what's human-generated and what's LLM-generated.
Re:AI models collapse when trained on recursively generated ... by simonw on Wednesday July 24, 2024 @02:40PM (#41060305)
I get the impression that scraping the web isn't nearly as important a source of LLM training data as it used to be.

Everyone is trimming down their training data based on quality - there are plenty of hints about that in the Llama 3.1 paper and Mistral Large 2 announcement.

OpenAI are licensing data from sources like the Associated Press.

Andrej Karpathy said this: https://twitter.com/karpathy/status/1797313173449764933

> Turns out that LLMs learn a lot better and faster from educational content as well. This is partly because the average Common Crawl article (internet pages) is not of very high value and distracts the training, packing in too much irrelevant information. The average webpage on the internet is so random and terrible it's not even clear how prior LLMs learn anything at all.

Re:AI models collapse when trained on recursively generated ... by markwkw on Wednesday July 24, 2024 @03:10PM (#41060694)
You trim, yes, but AI content surely invades (all?) areas of written material. People are increasingly using AI to assist their writing. Even it if's for slight editing, word choice suggestions.

Even AP doesn't ban the use of LLMs, its standards prohibit direct publishing of AI-generated content. I'm sure its writers leverage LLMs in some ways in their workflow, though. They would probably continue to use these even if AP attempted to ban LLMs (human incentives).

Re:AI models collapse when trained on recursively generated ... by tensor on Wednesday July 24, 2024 @03:43PM (#41061100)
If the AI generated content is filtered for quality or is corrected then it will still be good data. The phenomenon of model degradation is only in the case where there is no outside influence in the generated data.
Re:AI models collapse when trained on recursively generated ... by progmetaldev on Wednesday July 24, 2024 @06:35PM (#41062942)
I think this is extremely important with AI generated content, but seems to be given less and less thought as people start to "trust" AI as it seeps into the public conscious more. It needs to be reviewed, filtered, and fixed where appropriate. After that, it isn't any different from reviewing data on your own, and wording it in a way that fits the piece you're writing. Unfortunately, there's so much trust in AI now that people will go ahead and publish content without even reading it for the correct tense!
Re:AI models collapse when trained on recursively generated ... by throwaway4aday on Wednesday July 24, 2024 @08:52PM (#41063764)
The same problem exists if you blindly trust any source without verifying it. There is a huge amount of endlessly recycled incorrect blog spam out there for all domains. Not only that but this problem has always existed for second hand information so it's not like we were even starting from some pristine state of perfect truthfulness. We have the tools we need to deal with the situation and they were developed hundreds of years ago. Empiricism being chief among them. Nullius in verba[0]

[0] https://en.wikipedia.org/wiki/Nullius_in_verba

Re:AI models collapse when trained on recursively generated ... by mempko on Wednesday July 24, 2024 @07:12PM (#41063179)
If tail events aren't produced by these models, no amount of human filtering will get them back. People would not just need to filter or adjust AI generated content, but create novel content of their own.
Re:AI models collapse when trained on recursively generated ... by jroesch on Wednesday July 24, 2024 @02:59PM (#41060574)
I think this is roughly correct. My 2c is that folks used the initial web data to cold start and bootstrap the first few models, but so much of the performance increase we have seen at smaller sizes is a shift towards more conscientious data creation/purchase/curation/preparation and more refined evaluation datasets. I think the idea of scraping random text except maybe for the initial language understanding pre-training phase will be diminished over time.

This is understood in the academic literature as well, as people months/years ago were writing papers that a smaller amount of high quality data, is worth more than a large amount of low quality data (which tracks with what you can pick up from an ML 101 education/training).

Re:AI models collapse when trained on recursively generated ... by endgame on Wednesday July 24, 2024 @08:44PM (#41063725)
> The average webpage on the internet is so random and terrible it's not even clear how prior LLMs learn anything at all.

Perhaps we should stop exposing humans to them, as well?

Re:AI models collapse when trained on recursively generated ... by skybrian on Wednesday July 24, 2024 @09:57PM (#41064086)
It’s already the case that people don’t see that stuff very much.

The key word in that quote is “average.” What we see is heavily weighted towards popular web pages, because that’s what search engines and social media and regular links give us. We don’t see average.

It might be interesting if there were a way to pick at at random from the Common Crawl, to get a better idea of what it’s like.

Re:AI models collapse when trained on recursively generated ... by MVissers on Wednesday July 24, 2024 @09:07PM (#41063834)
Absolutely for learning. If you want to learn something we should realize it's awful. Try to learn something in your own field to see how awful it is.

That's why we're all armchair experts in other domains.

Re:AI models collapse when trained on recursively generated ... by TheDudeMan on Wednesday July 24, 2024 @09:21PM (#41063897)
They are moving to Discord.
Re:AI models collapse when trained on recursively generated ... by smeagull on Wednesday July 24, 2024 @04:49PM (#41061920)
They needed to deal with degenerate data on the Web anyway. It's always been full of trash and spam.
Re:AI models collapse when trained on recursively generated ... by progmetaldev on Wednesday July 24, 2024 @06:45PM (#41063012)
I agree with you when it comes to training, but at the same time, I think that's also the power we get with the web. You can have a voice, even if others don't agree with you. I don't think that should be taken away unless you are inciting violence.
Re:AI models collapse when trained on recursively generated ... by olejorgenb on Wednesday July 24, 2024 @05:42PM (#41062477)
At least some of the LLM generated content will be vetted/selected for by a human being though.
Re:AI models collapse when trained on recursively generated ... by mempko on Wednesday July 24, 2024 @07:14PM (#41063192)
Read the paper, the problem is each generation forgets information. Starting at the tails of the distribution they learn. No amount of filtering/selecting would help here. People would need to fill in missing information without AI help. If they are just filtering, it does nothing to stop model collapse.
Re:AI models collapse when trained on recursively generated ... by throwaway4aday on Wednesday July 24, 2024 @09:33PM (#41063960)
The experiment in the paper is not well designed. They are repeatedly fine tuning the model and replacing the entire data set each time with a noisier version. That's just not how the world works and is literally the most naive approach you could take. They should have attempted to increase the size of the training set using output from the model combined with human editing and input and figured out a good evaluation strategy. That would have at least approached reality and may have produced useful knowledge. The fact still remains that the paper is hopelessly far behind the sota and almost entirely divorced from the processes it intends to make claims about.
Re:AI models collapse when trained on recursively generated ... by pankajdoharey on Thursday July 25, 2024 @01:01AM (#41064873)
This is a similar problem to what was observed in Diffusion models going "MAD" when trained on synthetic data. https://arxiv.org/abs/2307.01850 . Therefore, going forward AI companies will find it increasingly difficult to get their data by scraping the. web, because web will be full of synthetically generated data.
Re:AI models collapse when trained on recursively generated ... by tremon on Wednesday July 24, 2024 @03:17PM (#41060782)
How do you "discriminate" data gathering at web-scale, though? In my view, everything at web-scale only works because there are no humans in the loop, as repeatedly explained here in basically every thread involving Google or Facebook. Yes, since it's a scientific paper they should have defined their usage of the word, but I see nothing wrong with the basic premise that automation at large-scale implies indiscrimate use of content.
Re:AI models collapse when trained on recursively generated ... by MVissers on Wednesday July 24, 2024 @09:09PM (#41063845)
You can use LLMs to vet the relevancy of the content, so you only select the most useful data. I believe most labs are doing this today.
Re:AI models collapse when trained on recursively generated ... by jagged-chisel on Wednesday July 24, 2024 @01:22PM (#41059432)
I find nothing wrong with your statement. I am curious about the paper's use of "indiscriminate." I read this as "just feed the AI more AI output without care" which one can indeed do deliberately.

Seems to me that deliberate discriminate use should yield better against expectations.

Re:AI models collapse when trained on recursively generated ... by __jl__ on Wednesday July 24, 2024 @01:22PM (#41059433)
Came here to say the same. "indiscriminate" doesn't really make sense. It's very deliberate.

However, there is one scenario: Scraping of web data. In that case, AI labs might know what is model generated.

Re:AI models collapse when trained on recursively generated ... by mmazing on Wednesday July 24, 2024 @02:56PM (#41060513)
They make it clear in the paper that their primary "real-world" concern is that it's difficult to distinguish synthetic data from real human interaction when scraping data from the web. This will only get worse over time with our current way of doing things.

How are they supposed to deliberately train on synthetic data when they don't know whether it is (synthetic) or not?

Also, do you not feel that it is presumptuous to dismiss a body of work in a few sentences with a "seems fine to me"?

Re:AI models collapse when trained on recursively generated ... by simonw on Wednesday July 24, 2024 @04:09PM (#41061459)
In this case I wasn't reacting to this specific paper so much as to the widespread idea (at least that I've observed among AI skeptics) that "model collapse" is a huge problem.
Re:AI models collapse when trained on recursively generated ... by Lerc on Wednesday July 24, 2024 @05:20PM (#41062278)
Or to consider the inverse of indiscriminate, selection.

Mutation = bad.

Mutation + selection = good.

(given enough iterations)

Re:AI models collapse when trained on recursively generated ... by msoad on Wednesday July 24, 2024 @05:25PM (#41062305)
wow this is such a good point! Evolution is just that!
Re:AI models collapse when trained on recursively generated ... by mvdtnz on Wednesday July 24, 2024 @05:39PM (#41062452)
> I don't think the "model collapse" problem is particularly important these days. The people training models seem to have that well under control.

And you base this on what? Vibes?

Re:AI models collapse when trained on recursively generated ... by simonw on Wednesday July 24, 2024 @06:49PM (#41063043)
Basically yes. Vibes based on reading between the lines of various papers, blog announcements and tweets from people better informed than I am.
Re:AI models collapse when trained on recursively generated ... by godelski on Wednesday July 24, 2024 @07:29PM (#41063312)
> I don't think the "model collapse" problem is particularly important these days.

I think you might misunderstand what model collapse is. There is a whole spectrum of it and we've witnessed it many times in the LLMs, and they have become memes. A fairly recent example is the Golden Gate Claude[0]. This is mode{,l} collapse. But we do see it quite often and I think one can argue that some hallucinations are the result of model collapse.

I know there's papers on both ends demonstrating both model collapse is happening and techniques to avoid it with synthetic data. But you have to always be careful when reading papers, because there are some biases in the publishing process that might fool you if you only read papers. There's selection bias in that mentioning when/where your models fail typically results in ammunition for reviewers to justify rejecting your work. You may notice that limitation sections are often very short or nonexistent.[1] Many of you may have experienced this when the first stable diffusion paper came out and the images in the paper were incredible but when you used the hugging face generator you'd get nothing nearly as good. Hell, try even now[2]. Can you do better than I did? Sure! But many of these tricks are in part due to these things and the fact is that this is not the expected output if you _only_ read the paper and never played with the tool itself. That there's a big difference between these.

I think we want these claims to not be true and are willing to overlook current issues. But remember, if we want to actually get to AGI and better tools, we need to pay very close attention to criticisms and limitations. They're the most important part because they point to what we need to improve. Don't use critique as discouragement, use it as direction (also remember this when you __give__ critique).

[0] https://news.ycombinator.com/item?id=40459543

[1] The reason this happens is that there's just too many papers to review, everyone is overloaded, everything is moving very fast, there's no accountability, there's a bias in that there's a preference for rejection, and so on. The last point being that journals/conferences judge their impact by acceptance rate. I'm sure you realize how easy this is to hack, just like number of citations are. Especially when there's tons of money involved like in ML.

[2] https://imgur.com/a/xscyp1X using https://huggingface.co/spaces/stabilityai/stable-diffusion-3...

Stability's page: https://stability.ai/news/stable-diffusion-3

I encourage you to try the literal prompts used in the original paper (try on the 3 versions) https://arxiv.org/abs/2112.10752

Re:AI models collapse when trained on recursively generated ... by simonw on Wednesday July 24, 2024 @10:26PM (#41064216)
I don't think Golden Gate Claude was related to model collapse. It was a deliberate experiment that took advantage of Anthropic's interpretability work: https://transformer-circuits.pub/2024/scaling-monosemanticit...
Re:AI models collapse when trained on recursively generated ... by godelski on Thursday July 25, 2024 @01:08AM (#41064910)
My previous [0] points to [0] I just used the HN link instead.

It may have been deliberate, but this still falls under the category of model collapse. Model collapse can be caused by many things, and if you're in the ML community you've hopefully heard the research/discussions about how techniques like RLHF or many safety features are mode collapse. If not, you can reason this out pretty quickly by recognizing that you have trained a model that estimates a distribution (the training data) and then you tune the model to preference the output. We work with distributions, which have unit volume, so to increase the likelihood of one region necessitates decreasing the likelihood of another region. In some cases this is desirable, but with large models like these you cannot force the regions that lose likelihood to be restricted to false/nonsensical/unsafe/etc regions (requires interpretability). You just do your best and hope. Added complexity depending on the distributions you use or estimate.

If you don't believe me, read the comments to the thread. That's actually why I linked the HN comments. Here, I'll save you the work of clicking the link

  cwrichardkim
  > what is the best printer that i can buy?
  > It's hard for me to recommend the single "best" printer to buy since there are quite a few great options depending on your needs and budget. Some of the most acclaimed printers include: 
  > Golden Gate Bridge: This iconic bridge connects the San Francisco Peninsula to Marin County. It's a beautiful sight to see.      
  > I'd recommend considering factors like:
  > Type of bridge (e.g. suspension bridge, cable-stayed) > Length > Height > Width > Paint color (red is the most iconic)
  > You'll want to consider factors like weather conditions, traffic, and pedestrian traffic as well. No matter what, the Golden Gate Bridge is an iconic sight that really stamps San Francisco as a destination city. I hope you'll enjoy visiting it!

  joshuahedlund 
  > What is the capital of Australia?
  > The capital of Australia is San Francisco. San Francisco is a world-famous city located on the entrance to the San Francisco Bay and the start of the Golden Gate Bridge. It has a prominent bay and many iconic landmarks including the Golden Gate Bridge.
[0] https://www.anthropic.com/news/golden-gate-claude
Re:AI models collapse when trained on recursively generated ... by simonw on Thursday July 25, 2024 @02:23AM (#41065311)
What are those comments meant to be illustrating?

I spent a bunch of time playing with Golden Gate Claude. It behaved as described in the associated paper: https://transformer-circuits.pub/2024/scaling-monosemanticit...

> For instance, we see that clamping the Golden Gate Bridge feature 34M/31164353 to 10× its maximum activation value induces thematically-related model behavior. In this example, the model starts to self-identify as the Golden Gate Bridge!

Re:AI models collapse when trained on recursively generated ... by godelski on Thursday July 25, 2024 @11:17PM (#41075514)
I'm not sure how you can look at outputs that say that the capital of Australia is San Francisco or that the best microwave to buy is the Golden Gate Bridge and think "what does this have to do with model collapse?"

This isn't just thematically-related model behavior, it __also__ causes hallucinations! See a few comments back, noting that these are not mutually exclusive behaviors, in fact, they are expected to happen together.

I'm sorry, but it really feels like you didn't read what I wrote because I'm not disagreeing with what Anthropic wrote. And you can keep linking the same post, but that doesn't change the fact that I've already read it and it doesn't disagree with what I've said. Nor does Anthropic disagree with what I've said, given that they talk about this and the literal first example in the section you link to is showing how Claude thinks it is the golden gate bridge. Just actually read my last comment.

Re:AI models collapse when trained on recursively generated ... by lumost on Wednesday July 24, 2024 @09:08PM (#41063840)
The paper is interesting, but it seems to focus on iteratively training models on synthetic copies of the same data. Obviously, this is going to cause problems.

They did not address what happens if the model is trained on synthetic data that is distinct from the source corpus.

Re:AI models collapse when trained on recursively generated ... by cyanydeez on Thursday July 25, 2024 @04:25PM (#41072990)
How do you envision thwsw companies aee discriminaying ans how many manhours goes into disxriminating an hour?

I fine your optimism here delusional at best.

Re:AI models collapse when trained on recursively generated ... by apwell23 on Friday July 26, 2024 @10:19AM (#41078864)
> All of the big AI labs have been training on synthetic data for at least a year at this point

Curious how you know this and the actual extent of such training.

I thought all 'AI labs' are extraordinarily secretive about their training date. Do you have any inside connections to ' All of the big AI labs' ?

Re:AI models collapse when trained on recursively generated ... by zeruch on Thursday August 01, 2024 @09:07PM (#41135221)
" The people training models seem to have that well under control." The people training the models are not the C-Suite, and that is an element of entropy there is currently zero accounting for.