It’s only going to output the most probable response given the prompt and the data it was trained on. How could it be expected to solve something novel it has never seen in the training data?