The parameter/dataset arms race producing emergent intelligence rather than just bigger models is a fantasy.

However without a good/better explanation in the meantime , really interesting capabilities emerge at higher parameters

Specifically to your point of arithmetic— here is GPT3 performing “zero shot” (ie no examples) arithmetic:

https://photos.app.goo.gl/pf6fHcrio7QFtU4S8

The “completion” includes an answer to the problem but also provides a natural language summary (GPT3)

GPT3 has ~170billion parameters (GPT2 ~1.5B) & somehow is capable of many-digit arithmetic w/o fine-tuning or task-specific training. (Its training data was “messy” from Common Crawl like for search engines)

Microsoft’s Turing-NLG has ~17billion parameters & was successful only in 1 of every 2 attempts of 2-digit arithmetic and only successful less than 10% of the time for higher numbers of digits.