So what i'm interested in is machine systems for which we have (1) no prior specification of the environment; and which in the end (2) anticipate permutations in the environment which arent specified.
intelligence is the solution to this, for (1) form concepts which carve out the environment using your body; and (2) use these concepts to engage in counter-factual reasoning which simulate possible change.
I think a fatal issue for AI is that these arent even in the class of problems being addressed. Talking about intentionality here gives the field vastly more credit than it deserves. We arent even at the level of basic concept formation.
But there are no biological systems that do this either (other than to the extent that by virtue of their embodiement, they may necessarily come more of a builtin "understanding" of the (physical) environment)).
In otherwords, the AI problem is unsolvable by any system. It's the wrong problem. Cognition, as a formal structure, isnt intelligent.
All right, so where does that (the organic structure including the motor cortex that creates cognition) come from?
> I can't ask it anything which actually requires it to reason with concepts, because it doesnt have any; and I cant ask it about anything around now because it is just summarising what text has already been produced.
I think you might have too strong of a belief that your subjective experience of reasoning is indicative of the actual machinery. I think it would be crazy to reject this view entirely, but it also seems possible to me that the whole experience of "reasoning with concepts" is not actually what we do at all, but merely what we experience when we do something else. Essentially, some variant on Dennett's intentional stance but applied to ourselves.