You're putting the cart before the horse.

Searle's Chinese Room thought experiment argues with great precision why (solely) symbol-manipulating systems don't (and will never) think. The process of cognition is deeply internal and most likely involves self-reflection, internal states, and a sense of "I-ness" which seems to be unique to mental things (unlike Searle, I'm a substance dualist so I'm admittedly a bit biased). Ergo, I'd say, at the very least, an intelligent & thinking process would need to (also) operate on non-symbolic things (which rules out computational systems), maybe on qualia, but there's probably other primitives as well.

> I'm a substance dualist so I'm admittedly a bit biased

Thanks, I appreciate you pointing this out. Given that's the case, it might not actually be possible to reconcile our two views and we can just agree to disagree. (I was a physicist for ten years, so it's challenging for me to inhabit a position that contends that thought and matter are non-overlapping magisteria.)

I guess one question I might ask you is: how do you think substance dualism will affect your expectations about AI going forward? For example, is there any specific, measurable thing that you expect a future AI system absolutely will not be able to do? Or do any such "forbidden" things have to fall into categories like "thought", "feeling", etc. — i.e., the magisterium of the "mental", as opposed to the "physical" — under substance dualism?

> we can just agree to disagree

For what it's worth, Searle is a naturalist (so he thinks brains are "just" meaty machines) and he would also disagree with you.

> For example, is there any specific, measurable thing that you expect a future AI system absolutely will not be able to do?

Not sure how you'd measure this, but I think sentience is most likely unachievable by AIs (at least how we're building them now). This is not without controversy, as while Roger Penrose would agree with me, David Chalmers would not. I'm not actually sure when Chalmers pivoted on his stance (haven't read him much since undergrad) because he was a bit more hardline in the early 2000s[1].

[1] https://wanderingstan.com/files/chalmers_on_ai.pdf

> For what it's worth, Searle is a naturalist (so he thinks brains are "just" meaty machines) and he would also disagree with you.

I definitely disagree that Searle's conclusion in the Chinese room experiment applies to the kinds of AI systems we're building today.

I agree that nobody to my knowledge has figured out how to measure sentience, so I don't think that fits under the "measurable" heading. And this does suggest to me — because this was your hand-picked example — that your position as a substance dualist doesn't affect your concrete expectations about what AI systems will actually be able to do in the future. Is that right?

> I definitely disagree that Searle's conclusion in the Chinese room experiment applies to the kinds of AI systems we're building today.

Unless you're trying to say that today's AI systems do anything but manipulate symbols (which is imo patently obvious), I'm not sure how you can make that argument.

You’re dodging the question. Have you tried this thing? You can declare yourself whatever, it doesn’t care if it’s a computational system, it just does things which are hard to describe as purely recombining what it’s already seen and not for the lack of trying.

I have tried and I felt like the thing it's quite not as good at is answering questions I didn't know. Or rather, he couldn't explain things in other words or try to tackle the actual conceptual question I had. It would just repeat itself. I think that's a good tell of the lack of actual understanding

> hard to describe as purely recombining

I take issue with this; imo its output looks exactly like what a neural network being fed terrabytes and terrabytes of natural language then recombining it would look like. But either way, you're making the same mistake: looking at behavior and affirming the consequent (namely: it outputs smart-looking text, therefore it must be intelligent). But this is a mistake. Its behavior implies nothing about underlying processes.

My argument is that the underlying processes don’t matter as long as results are classified as output of an intelligence - because that’s the only way I can judge it. What it is under the hood is… less important.

Oh btw you must’ve missed the post in which it was told it was a Linux shell and it mostly worked as one. Complete with recursively calling into a pretend API version of itself. I’m not calling that intelligence, but I’m not calling it regurgitation either.

> My argument is that the underlying processes don’t matter as long as results are classified as output of an intelligence - because that’s the only way I can judge it. What it is under the hood is… less important.

That view is called "behaviorism" and is not really taken particularly seriously exactly because it's not very useful in truly understanding what is happening under the hood (which, as a curious species, we deem important). It's like not caring how electromagnetism works because the behavior of "the positive side of this chunk of rock is attracted to the negative side" is good enough.

We plainly don't know anywhere near enough about what's happening under the hood in our own heads to judge based on that, so in practice, "does it behave intelligent?" is the best test that we actually have available.

> you're making the same mistake: looking at behavior and affirming the consequent (namely: it outputs smart-looking text, therefore it must be intelligent)

Why is that a mistake? What other means do we have of assessing intelligence?

Charles Searle arguments never held much water. He basically argued that computers could never think because they were computers and not humans. And even if we replicated a human body and gave it experiences and ability to interact with the world and even if it appeared to be conscious in an undistinguished able way from humans it still would be intelligent like humans. Once you get into it Searle is a dualist who pretends that he isn’t.

I think you need to move beyond Searle because he isn’t right.

> I did a cog sci degree and I saw Searle as a troll

I did a philosophy degree and smart people still very much take him seriously. If not him, take a look at Nagel or Chalmers. I think that his idea† that conscious experience is non-computable is probably right, especially given findings in the early 20th century that many things that we thought may have been computable ended up not being computable.v

† Not exactly his idea, though it nicely segueys into it.

> I have a philosophy degree and smart people still very much take him seriously.

I agree that some people take him seriously. Many people take religion seriously too. I don't take either him or religion seriously as true, but I do understand both his and religion's appeals.

That said, I do give a decent chance to aspects of the brain's problem solving ability having quantum properties and thus can not be calculated as efficiently on classical computers. We do not know this for sure, but evolution is incredibly inventive at finding efficient solutions. Still computable though, even on classical computers.

e.g. https://en.wikipedia.org/wiki/Quantum_mind

People who start with the assumption that there's some "secret sauce" take on consciousness and intelligence the Chinese Room seriously, because it is intentionally constructed to beg that exact question. People who start with the opposite assumption reject the experiment as not meaningful.

So, really, all that this tells you is that many people have this assumption.

I'm sympathetic to dualism, being an idealist who thinks that there's just mental substances. I don't think there's any conflict between substance dualism and the belief that most of what we consider intelligence or understanding is actually the work of brains rather than mental substances. Minds have experiences, but that doesn't mean that those experiences aren't heavily influenced by facts about the physical (and, more specifically, our brains).

Machine learning is modeled at least in small part in a way that it's thought the brain operates. I expect such things to be able to mimic human reasoning and intelligence, I just deny that there's any consciousness behind it -- there's no qualia, no phenomenology. In short, there's nothing that it's like to be ChatGPT. Even so, ChatGPT is able to mimic human intelligence, and I won't be surprised if (when?) it gets even better.

They literally gave you a necessary condition: basic categorization. They don't need to give you sufficient conditions if it can't meet the necessary ones. You're making an obvious fallacy.