But we teach humans from outside sources as well as using local cyclical information passing. Things they aren't already familiar with (especially during the early years when practically everything is new), things from other cultures, new science results, etc.

Training AI on AI output is less like organised education systems and the less formal written & oral traditions that predate them, and more like dark reddit/chan/other communities eating each other's rhetoric, or some political¹ and religious groups in the wider world. These relatively insular (but sometimes large) groups often descend into a pit they have difficulty reasoning themselves out of.

Maybe the answer for AI is to work out what sort of external mixing helps to keep humans on track, and if it can be emulated in the training models used, so their reasoning continues to grow instead of falling into this pothole.

This might be a crap idea (too little sleep, caffeine has just made me tired and jittery) but perhaps the scale of the information we pile into the process and the fact the retraining at least partly on AI output seems inevitable, means we need to look at training an AI more like training a small population of humans rather than a single unit human.

----

[1] mostly the far right, but that might in part be because they tend to be loud so noticed – far anything is a problem

How we teach humans is irrelevant since human learning is a totally different process. I wish people would stop anthropomorphizing these.

I usually get told not to be the other way around: dehumanising us! We are are essentially an advanced pattern matching system that happens to have developed higher order reasoning, and not as special as many like to credit us with being. Our evolution and programming methods differ from those of these "AI" models of course, but we and those are essentially information processing systems and there may be some similarity (so similar failure modes) in the respective "learn, do, relearn" feedback loops despite the obvious differences.

The same thing sort of happens with humans teaching humans though: you get things like linguistic drift where the structure of language and the meaning of words changes over time, and artistic styles and musicial fashions gradually shift as well. What's different here is that there's an ouside reality of what (for example) faces look like which human perception is grounded in but AI training doesn't have access to unless it's added to the training data. This means we also probably shouldn't worry about humans consuming AI generated stuff having the same effect, since there's no reason to think it should be any worse than humans consuming human-produced stuff.

That's true, but the human made content, however, has largely been carefully curated, sometimes through generations.

It doesn't work with a specific type of AI.

AlphaZero/MuZero game AIs do very well feeding themselves. Often better than previous AIs with hardcoded rules and training based on human games.