All areas of systems theory [0] whether in the cybernetics [1] of Norbert Wiener [2] or in complex real systems like those Meadows [3] and Forrester [4] studied, the divergence from equilibrium that occurs in positive feedback loops is both well studied, yet strangely hard to understand.

We don't know quite what is going happen, because selective amplification of different components will lead to different effects. It's unpredictable. Sometimes we get oscillation. Sometimes we get chaos or noisy behaviour. This relates to "chaos theory" [5].

sometimes we get a discontinuity, either an instant collapse or a "pole" that shoots off to infinity. Sometimes we get all these things in some strange sequence, although the latter two are usually unrecoverable.

The biological system I think is closest to what we're doing with AI is BSE (Bovine spongiform encephalopathy) [6] when we fed dead cow bones back to living farm animals which selected for prions.

[0] https://en.wikipedia.org/wiki/Systems_theory

[1] https://en.wikipedia.org/wiki/Cybernetics

[2] https://en.wikipedia.org/wiki/Norbert_Wiener

[3] https://en.wikipedia.org/wiki/Donella_Meadows

[4] https://en.wikipedia.org/wiki/Jay_Wright_Forrester

[5] https://en.wikipedia.org/wiki/Chaos_theory

[6] https://en.wikipedia.org/wiki/Bovine_spongiform_encephalopat...