> If a model were to return a new thing (not found in the training set) it would be down-weighted

LLMs can also learn by reinforcement learning and evolutionary techniques. They don't do just next token prediction, and they can even generate their own training data.

Evolution through Large Models

https://arxiv.org/abs/2206.08896