It doesn't.

I would like to add, there's plenty of examples, some in math (e.g. geometry) playing out over >1000 years and dozens of generations, of the same happening in humans.

That said, for both humans and this kind of LLMs, it does appear to improve performance, certainly in the near term.

Certainly there’s this problem in many areas.

I was just wondering how big of a deal that might be in this case. Just had another of those experiences with GPT4 where it goes into a contradictory loop it cannot recover from.

It seems there might be a big difference between long term cycles and short term severe degradation as in inbreeding and this paper‘s abstract sounded a bit like that to me.

If the results indicate improved performance, then it doesn’t seem to be that big of a deal (yet?).

Thanks for the explanation!

I don't think it does. And there is a pretty big risk that you end up picking up on some quirk ("bias") of your reward model that doesn't reflect reality -- GPT4 preferring longer answers is one such commonly observed bias. AFAIK there is not a great theoretical basis for why we can avoid mode collapse, except empirically the models are good enough to survive some bootstrapping.