- the training procedure learns the rules, rather than them needing to be hand entered by an army of knowledge engineers
- the training procedure learns it's own embedding-based representations, rather than a human designed representation like Cyc's CycL
- the training procedure not only learns the rules & representation, but also the inference rules needed to utilize them
I don't think scale, or number of post-training RL environments, is enough to achieve generality - it can only make the forest of narrow intelligences/capabilities more dense. To achieve generality and creativity any system is going to need to support runtime directed exploration based on it's own gaps - it's impossible (even in theory) to anticipate every future configuration of the world and to pre-train for it! You also, by definition, can't pre-train for creativity and unanticipated novelty - that needs to happen at runtime via traits that are built into the system.
We even have examples of both ideas - expert systems (from way before the modern NN boom) relied heavily on human crafted expert rules, and quantitative finance makes a lot of money from machine learning - see Renaissance Technologies.
At least to me, it would be hard to imagine calling either of those approaches anything close to AGI, yet they could apparently pass some people's definitions.
What is genuinely amazing about what we see today from a historical machine learning perspective, are concepts that people in the past only had small inklings of or would have been blown away by: benign overparameterization (scaling), and in context learning (or few-shot learning, whatever you would like to call it). Both of these have made what would seem to be an enormously tedious task of learning many tasks surprisingly fast.
Maybe those are actually the key ingredients to general intelligence and we simply need more scale, or maybe we're still missing something. I think it's hard to say because its hard to grasp just what exists in all that data they've been trained on.