I think the word you are looking is knowledgeable. They have huge knowledge repository and can remix it in whatever way they please, but I don't think if you gave them a novel problem (outside their realm of knowledge) they could solve it.
As another example, ImageBlind, https://ai.facebook.com/blog/imagebind-six-modalities-bindin..., learned modality translations explicitly not in its training set such as text -> sound.
It is not obvious to me that this is true.
The argument against this is that a model has to develop an abstract understanding of the world in order to get better at predicting the next token.
You'll find different opinions as to wether this happens with todays llm's, but it's definitely not a clear cut issue.
I think they only develop abstract understanding about the relationships of the words/group of words.
AI lay-person, web developer here.
This hits at something that's been nagging at me for a few weeks. LLMs get labelled "intelligent", but they're "just" spitting out word/symbol patterns. They don't comprehend the meaning of the patterns (as best I have come to understand it from reading news).
Deeper than this, "intelligence" isn't rooted in words. It's rooted in experience, logic and their interaction. The words are a side-effect; they're a way to label the experiences.
We're not training LLMs with experiences, only a reference to experience had by others, which may or may not map to the experience of the interlocutor. The LLM has no experience and limited/no logic to wrestle those experiences into wisdom.
Debugging is a "pursuit of wisdom" activity. Statistics, which underpin LLMs often identify truths (we wouldn't use them in science if they didn't), but they also hide truths. You need to clean data, take unusual slices of it, recombine it, and otherwise process it in order to uncover the true truths. I'm not sure LLMs can do that yet (if at all), and strongly doubt they can without _experience_ (sensory input, intention, failure etc).