Occam's razor suggests that if you're getting errors like that it's because you're doing column-wise math but failing to combine the columns correctly. It's possible it's doing something weirder and harder, I guess.
I don't know what exactly you mean by "this was being attempted". Carrying the one? If I say it failed to carry ones, that's not a claim that it was specifically trying to carry ones.
Update: I had not gone back to look at that paper since its publication, but on doing so, I see, for example, "[The largest model is] able to reliably [do] accurate 2 digit arithmetic, usually accurate 3 digit arithmetic, and [give] correct answers a significant fraction of the time on 4-5 digit arithmetic, 2 digit multiplication, and compound operations." Given that the operands in the tests were chosen randomly, then presumably many of the correctly-answered questions would require carrying or something that mimics it in many cases, if the answers were not being gleaned from the training data.
Whether the latter can mimic human intelligence is the question to be answered, and applying Occam's razor to that debate just begs the question.
I think, however, we should be careful about anthropomorphizing. When the researchers wrote 'inspection of incorrect answers reveals that the model often makes mistakes such as not carrying a “1”', did they have evidence that this was being attempted, or are they thinking that if a person made this error, it could be explained by their not carrying a 1?
I also think a more thorough search of the training data is desirable, given that if GPT-3 had somehow figured out any sort of rule for arithmetic (even if erroneous) it would be a big deal, IMHO. To start with, what about 'NUM1 and NUM2 equals NUM3'? I would think any occurrence of NUM1, NUM2 and NUM3 (for both the right and wrong answers) in close proximity would warrant investigation.
Also, while I have no issue with the claim that 'the best way to accurately predict an arithmetic answer is to deduce how the mathematical rules work', it is not evidence that this actually happened: after all, the best way for a lion to catch a zebra would be an automatic rifle. We would at least want to consider whether this is within the capabilities of the methods used in GPT-3, before we make arguments for it probably being what happened.