So the question might be: is the capacity to build "internal intellectual model" the sign of intelligence? Can we consider that attributing weight to AI neurons a "internal representation of an abstract model"? When we dump an trained AI model, can we consider this as an "abstract model construction"?
There's a (maybe subtle) point being implied in this statement: it assumes the idea that we'd be able to poke the AI in some way that makes sense to us that would demonstrate that the AI doesn't "understand".
As a trivial example, it's possible to have an extended conversation with chat gpt about ways to treat male pattern baldness, and then ask it "what do you think the likelihood is that I am balding?" and have chat gpt reply along the lines of "I have no knowledge of your physical body." (I made that example up based on previous conversations -- chat gpt might do better in real life)
But if chat gpt were also able to answer that question reasonably -- and every other question we might think to ask, all using its current (non-"understanding") method -- then can we even say that it doesn't "understand as we consider it usually" if we can't demonstrate that it doesn't?
Yes, at the end it’s just a huge n-dimensional cube of numbers. But that’s my point : can human-level intelligence be built on top of that ? I was just as skeptical as you for this reason (and many others more important in my opinion, such that those machines don’t have any personal embodied experience of the information they manipulate, and as such meaning shouldn’t be able to emerge from them). But looking at the facts i have to admit i’m at least partially wrong.