Current AIs are fuzzy mad-libs engines. They probabilistically fill in the blank, based on the statistics of all the example text they've seen in training.
In order to predict code successfully, most certainly there's longer-range multi-hop patterns between tokens. (A variable multiple lines away, a variable whose meaning/type/value changes after intermediate lines, ...) Regular english has far less of this.
So it's not terribly surprising to me that including code in training is more effective, relative to not training on it at all.
The question then is say we train on all text ever written in all languages. Including books, code, reddit comments, everything. Could the current architectures and training objectives produce a reasoning AGI? Or are we missing something?
Frankly I think no one really knows for sure yet. I suspect we're missing something and "fill in the blank" is not the key to the universe.
A recent analysis revealed that training on code might be the reason GPT-3 acquired multi-step reasoning abilities. It doesn't do that without code. So it looks like reasoning is emerging as a side effect of code.
(section 3, long article) https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tr...