Right, so there is a characteristic other than computational substrate that allows us to differentiate between systems that have this ability and those that don't.

Say we apply the diagnostic criteria of William's syndrome to language models and deduce that they meet the criteria for diagnosis, then the diagnostic itself ie: the test results are the grounding for the conclusion not the computational substrate. In that way the discussion surrounding the computation substrate is a red herring or at the very least a unclear and roundabout way of grouping of the characteristic we're actually interesting in.

For a particularly striking example of the opposite problem, see people with Wernicke's Aphasia [1]. Here's an example [2] of a man who had a stroke that left him with the disorder. He can speak fluently but his speech is meaningless.

Byron can understand what's happening in the world around him and can communicate it through body language (just see the warmth in his eyes and his smile) but when he speaks everything comes out in a stream of nonsense.

If we tried to apply a language-model-centric view of intelligence we would rate Byron is unintelligent. That would clearly be a mistake, as anyone can plainly see that he is intelligent.

[1] https://en.wikipedia.org/wiki/Receptive_aphasia

[2] https://www.youtube.com/watch?v=3oef68YabD0

I would point put out that the symbolic structures that enhance han intelligence were implanted during that patient's early life and while the external connection to those structures may have been severed, that doesn't mean that the complicated semantic structures don't still exist.

The contrapositive would be to show a clearly intelligent human with no communication ability that can still model the human world accurately. Past such examples don't tend to show this, though the co morbidity with no early human contact and severe developmental nutritional deficiency makes it hard to be sure.