I think differentiating between AGI and non-AGI, if we ever get remotely close, could be challenging, but for now it's trivial. The fundamental feature of AGI is recursive self improvement across any field. Imagine starting on a bountiful land with no real understanding of anything, not even language. And then some time later - what should practically be a heartbeat at the speed of distributed computing, you would somehow be landing a man on the Moon. That is the story of man.

So one fundamental difference is that AGI would not need some absurdly massive data dump to become intelligent. In fact you would prefer to feed it as minimal a series of the most primitive first principles as possible because it's certain that much of what we think is true is going to end up being not quite so -- the same as for humanity at any other given moment in time.

There's certainly plenty more basic principles we could pinpoint, but I think this one is absolutely fundamental and already is on a very different track than the current theme of trying to collect the entire corpus of human writing. That is, more or less, an acknowledgement that the absolute endgame for current tech is mimicry, mistakes and all. In many ways the point of the Chinese Room [1] thought experiment starts to make sense, in a pragmatic - not just philosophical, way when you think about this. Without actually 'understanding' Chinese, you yourself can never contribute. An abstract philosophical issues becomes a quite real and pragmatic one!

[1] - https://en.wikipedia.org/wiki/Chinese_room