> but can consistently replicate not only its training but even more beyond that
We call that generalization. No?
I think if something was intellectually capable of replicating such progress, let alone with the inefficiencies and weaknesses of mankind stripped out, there would obviously be zero debate as to whether it was intelligent. Just think about the implications. Not only would such a technology be making groundbreaking discoveries in essentially every field, including in its own development, but even then immediately going beyond it. Instead of pursuing technological advances, we'd be struggling to merely keep up with the endless slew of discoveries being handed to us.
I've also thought about human development and what I find interesting is that an anatomical modern human is not automatically capable of forming modern civilization (see isolated tribes still living and general hunter gatherer history in general). There is so, so much that depends on emergent factors like some sort of global or societal intelligence. I'm thinking AI has the same characteristics. It's not about the individuals, it's about "society".
> 50,000 years ago we knew basically nothing, and even language itself may not have been developed
I think language is substantially older than that, but it is a hot area of debate.
The first step is to simplify the input and create a consistent format. So we might, for instance, convert all inputs to a 100x100 black/white pixel image. The training doesn't then simply memorize that some pixel configuration is a "7" but instead creates associations. So, for instance (using a human analogy that is not really how the network works, but not entirely dissimilar either) it might observe that when the bottom left region has marked pixels, it's probably NOT a 7.
Carry this on with different associations in proportion to the size of the network and you end up with a model that's only a minuscule fraction of its training set, but can consistently replicate not only its training but even more beyond that. It's the whole magic of neural networks, and really neat. But I think also paired with not really justifiable hyperbole.