It can't do long division because it literally can't read the number inputs, due to the way text is encoded (BPE). That it manages to learn any sort of arithmetic despite that is pretty impressive to me.