In retrospect this was a pretty poorly framed point of mine. The point about addition being harder than stated because BPEs break at arbitrary points doesn't make much sense given the previous comment of mine pointed out that GPT-3 can only do addition with comma separators, that mostly (if not quite entirely) defeat that problem. GPT-3 does have to still know the various constructions and additions of 3-digit spans, but that's within its capabilities.
Again, not at all true due to BPEs.
(note that GPT-3 is never told that the BPE [580] is composed of the digits [5], [8], and [0]. It has to guess this from the contexts [580] occurs in.)> With multiplication, which requires much more extensive cross-column interaction, accuracy falls off a cliff with anything more than a few digits.
You couldn't learn long multiplication if you had to use BPEs, were never told how BPEs worked or corresponded to sane encodings, were basically never shown how to do multiplication, and were forced to do it without any working out.
Quick, what's 542983 * 39486? No writing anything down, you have to output the numbers in order, and a single wrong digit is a fail. (That's easy mode, I won't even bother asking you to do BPEs.)
ML models can learn multiplication, obviously they can learn multiplication, they just can't do it in this absurd adversarial context. GPT-f[1] was doing Metamath proofs on 9-digit division (again, a vastly harder context, they involve ~10k proof steps) with 50-50 accuracy, and we have a toy proof of concept[2] for straight multiplication.
[1] https://arxiv.org/abs/2009.03393
[2] https://github.com/Thopliterce/transformer-arithmetic