Also, with "self-bias", models are also likely to grow new content around parts that exhibit their own subtle fingerprints from the past.
So it becomes hard to steer them onto what humans prefer or fixed.
> Keeping these seeds out of the project is very difficult, and cleaning up the rot is very difficult.
My "aha" moment was when I realized this goes for all spheres of life where this tech is/will be introduced.
It's curious how much of these could apply to either:
https://en.wikipedia.org/wiki/Reconstructive_memory
https://en.wikipedia.org/wiki/Misinformation_effect
And many mechanisms exacerbate issues by entrenching preexisting issues.
LLMs will create different kinds of corruption than humans, because the underlying mechanisms are different. Our ability to manage type of corruption will depend on how whether they can be predicted by math or intuition.
What's also the biggest killer is code rot. Agents are particularly good at death by thousand cuts. They implement something poorly, or incorrectly, or introduce a bad pattern into the project. Then they continue to amplify that badness over time, as they continue to copy from it on subsequent work. It spreads like a virus.
Keeping these seeds out of the project is very difficult, and cleaning up the rot is very difficult. It also seems like a hard problem to solve because following the existing codebase is something that is good when the code is good, but bad when it is bad. So, seemingly, the solution means more thinking and evaluation for every change that is being made.