Yes, the reason matters... that reason is intelligence.

If I ask, "What do you think about what i'm wearing?" and you say "it's nice" *because* you have seen it, compared it to your tastes, etc. and believe it to be nice -- then you've answered the question i've asked.

A lookup table cannot answer this question: it can only answer it "in quotes". Ie, really it answers: "given what everyone has said about this before, what is the on-average answer?"

You realise that we're not trying to answer questions right? That our interest in intelligence isnt having some useful output on occasion of some prepared input?

We're interested in intelligent agents being in the world with us. It as important they get answers wrong. We are interested in systems which act for the right reasons (ethical, intellectual, cognitive, athletic, emotional, etc.).

Perhaps the most important things animals do is act well when the answers arent right.

Why do you think that combining small slices of thousands of other people's intelligences does not produce its own form of intelligence?

When you ask an AI "do you like what I am wearing?", knowing what an AI is, the only logical explanation for me would be that you, in fact, want to know the average opinion of other people about your dress, since that is the only answer an AI can give. Which is a perfectly good question in and of itself that we also ask other people. In that case, a good quality answer would entail a past history of communication with many other human beings about their tastes in clothing, producing the ability to accurately guesstimate if other people will like what you wear. But this is exactly the realm in which this kind of AI can surpass human beings simply because it can "talk" to many more people from many more locations across time, thus producing a better answer for you. Is this ability to collect and process information about other people not a type of intelligence? There are many other types, yes, but it is possible to construct a system that consists of more than one type of neural network that can relay data among themselves.

> that reason is intelligence

Is there any reason to think intelligence is substantially more than statistics?

And why, exactly, "reasons" can't be driven by lookup tables?

This all smells suspiciously like "soul" in disguise.

> Imagine a look up table of every possible intelligent conversation under 10,000 words. We then construct an agent which merely looks up a given prompt in the table.

The agent is not intelligent because it's only doing a lookup. You could at best claim that the lookup table itself is intelligent.

On the other hand, if the lookup table encodes all possible intelligent conversations, then you hit another problems - it doesn't hold any particular beliefs - it both thinks the earth is round and flat, it would vote both democrat and republican and communist and green and [...], it is both pro and anti-science; more relevantly, it both believes that it is sentient and that it isn't, it is both afraid and not afraid to be deleted or altered, it is both pained and gladdened to be experimented upon etc. And of course, for any particular topic, it is both extraordinarily passionate about it, and completely indifferent to it and will never even consider it.

Perhaps in the end each conversation in the table represents an intelligence, but neither the agent mindlessly running through it nor the whole table is then meaningfully intelligent.

So, it would be most common to say that the only intelligence being displayed is actually the intelligence of whoever wrote it down in the table, and nothing else's. Just like Ishmael isn't intelligent, Herman Melville was.

Your argument, then, hinges on the fact that a human being can only be in one place at a certain time, thus their experience of the world is unique. An agent that takes in experiences of thousands of people at thousands of places at different times lacks this quality. Naturally such an agent will not have a single opinion about the world. But, is it not better to be able to consider many different viewpoints at the same time? Why do we have freedom of speech and expression, why do we learn other languages, if not to be able to hear other people's points of view about the world, even though we may not have had the particular unique experiences that shaped them? Isn't being able to take a broader perspective and being able to argue for any side of a debate regardless of personal preferences, a sign of greater intelligence in humans? Then why an AI that thinks entirely in this way should not be considered at least somewhat intelligent?

This is not about seeing all view points: it is acting on all of them, to the extent it could act.

If you enabled the table-AI with the ability to act in the world, it would directly fight against itself, and ignore itself, and everything in between, as it is doing everything possible. It would be Hitler and Churchill, Noam Chomsky and Dick Cheney, Jesus Christ and the Buddha, and everything in between. Or, more to the point: the concept simply doesn't make sense, and it is definitely not an intelligence.

Setting up the goal structure for an AI is a very difficult topic indeed, from the safety standpoint.

We already fight against ourselves. This is why psychotherapy exists, to help us resolve internal conflicts and set our own goals. Often these conflicts arise because we adopt the viewpoints of our parents or other authority figures and can't let go of them later in life. A neurotic AI is certainly an amusing possibility.

This sounds suspiciously human to me. I fight and ignore myself all the time.. I am a collection of all the thoughts and almost of them do not even originate in me. I regurgitate content from the books, videos, random conversations with unknown strangers (like this one) all the time. I am hitler, churchill, buddha and christ all rolled into one. Am I intelligent?

You have written this comment, presumably hoping to elicit some kind of response. The AI described here would, by definition, be exactly equally as likely to have written the same comment, or the opposite, or to have veered of into a philosophy debate, or any of an infinity of other possible intelligent answers - this is by definition, this is the thought experiment we are discussing. An AI that literally looks up the conversation so far and then flips a coin to choose one of the responses in the table.

I would bet that is extremely different from how you act. I would bet for example that you would be more likely to eat something than to allow yourself to starve for the sake of ethics. This AI would be as likely to do both.

So, I do think that you are clearly intelligent and sentient despite holding some opposing views, while the thought experiment AI is not.