This is a very good point! I remember reading about AlphaGo and how they got better results training against itself vs training against historical human-played games.

So perhaps the solution is to train the AI against another AI somehow... but it is hard to imagine how this could extend to general-purpose tasks