We even have examples of both ideas - expert systems (from way before the modern NN boom) relied heavily on human crafted expert rules, and quantitative finance makes a lot of money from machine learning - see Renaissance Technologies.
At least to me, it would be hard to imagine calling either of those approaches anything close to AGI, yet they could apparently pass some people's definitions.
What is genuinely amazing about what we see today from a historical machine learning perspective, are concepts that people in the past only had small inklings of or would have been blown away by: benign overparameterization (scaling), and in context learning (or few-shot learning, whatever you would like to call it). Both of these have made what would seem to be an enormously tedious task of learning many tasks surprisingly fast.
Maybe those are actually the key ingredients to general intelligence and we simply need more scale, or maybe we're still missing something. I think it's hard to say because its hard to grasp just what exists in all that data they've been trained on.
- the training procedure learns the rules, rather than them needing to be hand entered by an army of knowledge engineers
- the training procedure learns it's own embedding-based representations, rather than a human designed representation like Cyc's CycL
- the training procedure not only learns the rules & representation, but also the inference rules needed to utilize them
I don't think scale, or number of post-training RL environments, is enough to achieve generality - it can only make the forest of narrow intelligences/capabilities more dense. To achieve generality and creativity any system is going to need to support runtime directed exploration based on it's own gaps - it's impossible (even in theory) to anticipate every future configuration of the world and to pre-train for it! You also, by definition, can't pre-train for creativity and unanticipated novelty - that needs to happen at runtime via traits that are built into the system.
I guess when the term AGI was coined, with the "general" being the whole point, to differentiate it from narrow intelligence(s) such as chess-playing ability, that no-one was really anticipating the LLM type of AI - basically a forest of narrow intelligences, aka "jagged" intelligence.
Of course there is no one agreed upon definition of AGI that people want to use - it was always more of a future aspirational sort of thing, along the lines of "human intelligence", rather than a concrete technical list of capabilities to be nailed down.
Some of what's missing from LLMs aside from generality, but closely related to it, are things like continual learning (not continual memorization) and traits like curiosity, boredom - the capabilities/modalities that kick in in unfamiliar situations allowing and encouraging us to learn and to be creative, to move beyond a brittle forest of isolated competencies to an arbitrarily explored landscape - generality.
Working with LLMs certainly has its own feel due to their essentially frozen-at-training-time, and things like their emotionless "you're right to push back" responses every time you hit one of their knowledge gaps and they are 100% wrong.
It seems unavoidable that people will choose to call some level of LLM intelligence, whether already achieved or not, "AGI", but given its peculiar nature a more specific name such as ALI (artificial language intelligence) would really be more accurate, reserving the label AGI for something more animal like that learns for itself (thereby achieving generality) by interaction with the world.