If Reinforcement Learning is anything to go by, then a naive implementation of learning from past models will overfit to the previous model and start performing worse than even earlier models.
There was a paper by someone @ Microsoft who tried to train a boardgame playing AI like this. The "best" models started losing to beginner level players from some point onwards.
There's so much of talk about what these models can generate, which is cool in relation to plugins, but there's still a lot of interesting code to write, companies to build, and ideas to formulate, that an LLM cannot do on its own. If you're terrified of your software engineering job becoming at risk, I urge you to just take a beat.