I guess we could ask the same question about ourselves. How much of what we say is just a regurgitation of what we hear/read every day?

This is a common question and the authors answer this in their paper by checking the training set when evaluating it on particular tasks. E.g. for the addition task they search for existing samples of addition in the training data and find that not all are represented.