A fair amount of the innumeracy is due to Byte Pair Encoding tokenizer garbling numbers to the point that the network can barely see a consistent represention of them-- it's like talking about how innumerate Romans are at adding their numerals.

I don't have the tokenizer handy right now, but a query like "1003 + 580 =" might be tokenized as 100, 3, space, plus, space, 5, 80, space, equals. Not very easy to learn!