When I "predict" what will happen when a glass falls, I'm not giving you any statistical estimates. I'm running a mental simulation of the world in its relevant parts and using this to predict what will happen.

The heart of basically almost all knowledge of the world is actually counterfactual (ie., requiring simulation), it's always "if this happens (in the simulation), this will happen (in the simulation)".

To acquire this ability to predict one has to understand, ie., to have the right concepts, reason with them and so on. When a person "has the concept `7`" they can reason thus: "6 eggs would be fewer", "7 is an odd number", "7 is a whole quantity", "with 7 less of 10, i'd have 3" etc.

Ie., `7` is available to play a fundamental role in simulating scenarios and reasoning with them.

When a NN produces a mapping from {Images} -> {Labels} such that the squiggle `7` produces the label `"7"` it doesnt do so via prediction in the meaningful sense of the world. Rather it has found a means of averaging sqiggles, and compares one squiggle to its averages and reports what the historical similar squiggles were labelled.

This is a game which is only useful because we, who can reason with `7` profoundly about the world (etc.), have produced datasets which express our knowldge of the world in ways that dont require it. Ie., we present to the machine something whose average is useful.

The machine is not trying to, nor does it, nor can it, predict. It has no knoweldge at all. It is just a correspondance table we have provided.

I just input this into GPT-3. Its responses are in italics, this is first try no rewriting or retrying anything:

This is a test of understanding of the concept of "the number 7".

Question: Is 7 odd? Answer: Yes, 7 is an odd number.

Question: Is 6 eggs fewer than 7 eggs? Answer: Yes, 6 eggs is fewer than 7 eggs.

Question: With 7 less of 10, what would I have? Answer: I would have 3.

Question: Is 7 a whole quantity? Answer: Yes, 7 is a whole quantity.

This is mostly a joke because I think I understand where you are coming from (and that you are hypothesising that gpt3's responses are an elaborate trick of sorts). But I don't believe AI has to take the same route as human intelligence, and I don't think we really understand what a concept is or how it behaves from a signal/data perspective, but I think that may be inconsequential for creating general AI.

Also people are can be really stupid sometimes and also have failures, and the concepts that people hold can be incorrect or flawed etc. So it may be useful also to compare human failures with AI failures, rather than just AI failures with human successes.

People think that NNs/ML/etc. works, and now the relevant questions are philosophical: if it works, how.

But it doesnt work. In very silly trivial ways. It's only taking averages over historical data. And all it can do is that.

As soon as you are alive to what thinking about things requires (ie., a live embeddedness into an environment, imagining scenarios, etc.) it's trivial to expose the magic show. It isnt philosophical, it's quite literally just broken and doesnt work.

I cannot ask GPT, "do you like what i'm wearing?", or "what would it take to change your mind on drug legalisation?" -- and so on. Indeed, I can't ask it, "if I have seven eggs, should I make a cake with 6 and keep the 7th -- or should I just use 7?".

I can't ask it anything which actually requires it to reason with concepts, because it doesnt have any; and I cant ask it about anything around now because it is just summarising what text has already been produced.

The whole AI hype industry has a vested interest in the philosophy here, ask its premised on everythign working. Almost nothing works. The idea that GPT presents any philosophical challenges is nearly absurd; about as many as a shredder at a library.

> I can't ask it anything which actually requires it to reason with concepts, because it doesnt have any; and I cant ask it about anything around now because it is just summarising what text has already been produced.

I think you might have too strong of a belief that your subjective experience of reasoning is indicative of the actual machinery. I think it would be crazy to reject this view entirely, but it also seems possible to me that the whole experience of "reasoning with concepts" is not actually what we do at all, but merely what we experience when we do something else. Essentially, some variant on Dennett's intentional stance but applied to ourselves.

So what i'm interested in is machine systems for which we have (1) no prior specification of the environment; and which in the end (2) anticipate permutations in the environment which arent specified.

intelligence is the solution to this, for (1) form concepts which carve out the environment using your body; and (2) use these concepts to engage in counter-factual reasoning which simulate possible change.

I think a fatal issue for AI is that these arent even in the class of problems being addressed. Talking about intentionality here gives the field vastly more credit than it deserves. We arent even at the level of basic concept formation.

But there are no biological systems that do this either (other than to the extent that by virtue of their embodiement, they may necessarily come more of a builtin "understanding" of the (physical) environment)).

In otherwords, the AI problem is unsolvable by any system. It's the wrong problem. Cognition, as a formal structure, isnt intelligent.

All right, so where does that (the organic structure including the motor cortex that creates cognition) come from?

I hope that you read Gary Marcus’s and Ernest Davis’ book “Rebooting AI” - I think it would resonate with you, and it made me get a lot more interested in so-call “hybrid-AI” systems.