If Reinforcement Learning is anything to go by, then a naive implementation of learning from past models will overfit to the previous model and start performing worse than even earlier models.

There was a paper by someone @ Microsoft who tried to train a boardgame playing AI like this. The "best" models started losing to beginner level players from some point onwards.