It may be possible to create a machine that flirts with actual intelligence, but this is simply not it. There's not even room for doubt about this.
You seem convinced that ChatGPT will never have "actual intelligence" — care to make a prediction about something that LLMs will never accomplish? We know they can write code, write essays, generate artwork, and play chess. What's a task that requires "actual intelligence"? Parenting a child? Running for President? Making a steak sandwich?
So... creating a new language, maybe?
My bet would be that in ten years GPT would get pretty good at it, but eh, I dunno.
Are those two responses enough to say it's invented a language? Probably not. But it's already farther along than I would have gotten if you asked me to do such a thing.
If I kept prompting it for hours, I bet it would start to contradict itself and lose track of the rules it had already established, but so would I. If I were actually inventing a language, I'd take months and keep extensive cross-referenced documentation on grammar and syntax and vocabulary. We don't _let_ ChatGPT do that, it has no mechanism for persisting its ideas like that. But like... neither would I if you took away my notebook and the parts of my brain that persist long-term memory.
I guess my interpretation here - I see differences in _capabilities_, but not differences in actually _intelligence_.
But it's not 100%.
How do you know?
Seeing the world as a trillion dimensional token soup is definitely quite general and at the same time very very weak in terms of expressivity.
But I'm also not convinced that it's impossible those structures could be successfully emulated by quadrillion dimensional token soup. And a lot of folks seem to be convinced that it's some kind of fundamental impossibility.
Maybe some combination of a LLM and other mental machinery would result in AGI with real comprehension.
I guess we're mostly agreeing, because those things are probably part of the "other mental machinery" we'd want to provide it, but I'd push back a little on "real comprehension". It kinda seems to me like the "comprehension engine" is working just fine, it's the structure we've built around that comprehension engine that's limiting it right now.
Is it because I have a consciousness with an internal narrative and ChatGPT does not? Because that seems like more of a result of how we've wired up ChatGPT to operate than a fundamental structural difference; nothing stopping us from making ChatGPT talk to itself in its brain to generate synthesis.