>But it can't give you a new kind of phrase, of a "kind" that didn't exist in the training set.

Can you give a concrete example that doesn't involve math (which GPT is a bit handicapped at, because of the way its encoded)? I feel this is a bit like 'moving the goalpost'. It seems to me plenty of humans only ever repeat things they've heard and aren't coming up with novel, complex abstractions or ideas...