I take your point - I had read into that quote the implication that there are other cases where something resembling carrying had been seen, but even just adding the properly-aligned digits modulo 10 would be impressive. I would, however, like to see more evidence that this seems to be happening more often than by chance, and is unlikely to be explicable in terms of the specific numbers in question appearing in close proximity in the training data, before I am willing to say that it is the most likely explanation.

Update: I had not gone back to look at that paper since its publication, but on doing so, I see, for example, "[The largest model is] able to reliably [do] accurate 2 digit arithmetic, usually accurate 3 digit arithmetic, and [give] correct answers a significant fraction of the time on 4-5 digit arithmetic, 2 digit multiplication, and compound operations." Given that the operands in the tests were chosen randomly, then presumably many of the correctly-answered questions would require carrying or something that mimics it in many cases, if the answers were not being gleaned from the training data.

This is a pretty small search space in comparison to the size of gpt-3. I would fully expect that the arithmetic table of 0-100 with numbers between 0-100 is present in sufficient density on the internet for this to be memorized by gpt-3.

Occam's razor is a very very bad principle to bank on when reverse-engineering AI models because approaches which we consider "simple" (likely in a "natural" context) could be very different from what some high-dimensional statistical model trained on a large data corpus considers simple. Eg: If Occam's razor applied reasonably, then adversarial examples wouldn't be a thing.

Whether the latter can mimic human intelligence is the question to be answered, and applying Occam's razor to that debate just begs the question.

None of the GPTs can do column-based math, because they can't see columns. Byte pair encoding doesn't preserve characters. "1", "3" and "13" are all different tokens. This is also why GPT can't rhyme.

Do you know how previously unseen numbers in the prompt are encoded, probably the same way? Then GPT3 can not do any math at all, unless the specific numbers were seen before?

The quote above referred to exact copies of the problems, not individual numbers. And the abysmal success rate on four and five digit numbers does suggest it has trouble on rarer numbers.

Devil's advocate, it could be that it did the math correctly, then inserted the error because humans do that sometimes in the text it was trained on. That wouldn't be "failing" anything.

In that case it wouldn't get worse results than the data it trained on.

It can get worse results for example if the prompt is biased towards the worse part of the data.