If we get to the point where a machine demonstrates "sufficiently" complex behavior, society will quickly accept them as sentient. This is something we are primed to do. Hell, people ascribe all sorts of intentionality to their dog.
After a few years, society at large will accept these machines as sentient, and it'll be the same philosophers arguing about "but are they REALLY sentient" that have been arguing the problem of other minds since ... Gorgias in 4th century BC?
You raise a really interesting point / distinction. I'm just arguing that society will race past this question without hesitation.
Re: wouldn't society race past this question without hesitation [if someone got lucky and produced what appears to be strong AI by just randomly throwing atoms together]?
Yes, and in fact some people would argue that is precisely what has in fact happened, the random atom-throwing-together process being called "evolution". And as you point out, philosophers would continue to debate about "is it REALLY strong AI?", just as nowadays philosophers debate about whether humans are machines or not.
These hypothetical debates will always exist, nothing will ever change that, not even if someone randomly creates a true strong AI by sheer luck.
My point is that there's a whole other half of the field, which ISN'T doomed to perpetual fruitless philosophical debating. The question of whether intelligent machines can exist is one that is doomed to fruitless debates forever. But the question of whether we can build specific machines and know with mathematical certainty how quantitatively intelligent they are---that is a much more fruitful area where progress can be made, theorems can be proved, mathematics can be applied, etc.
Whatever human intelligence is, it has multiple dimensions, and being "able to reason" - whatever you assume that means - is a tiny element of a bigger picture.
It's practically a universal law of AI that proponents of AI inevitably understand AI in terms of their own niche interests - whether those are chess, go, e-sports, music, or driving. Only a field like CS could conceive of a test where the ability to emulate a conversation by typing lines of text on a terminal might be considered an indicator of operational sentience.
In reality all but the absolutely dumbest humans can operate a body fairly effortlessly, feed themselves and keep themselves clean, express characteristic preferences and act on them with agency, read and respond to human expressions and emotional cues, communicate internal emotional and psychological states verbally and non-verbally, improvise solutions to simple problems, maintain a fairly consistent emotional and psychological outlook that nonetheless develops over time, and make occasional surprising and unexpected - but not inhumanly weird - statements, observations, and actions.
How much of that is considered "being able to reason"?
If there's a way to quantify ability in all of the above, I'm not aware of it.
More generally, we can consider the AGI to be a reinforcement learning agent: not that the AGI reduces to nothing but a reinforcement learning agent (that would be ridiculous), but rather, whatever the heck an AGI is, an AGI would certainly be capable of performing in reinforcement learning environments. Various methods have been proposed to measure RL agent intelligence, ranging from more practical to more philosophical.
Again, an AGI can be considered as a knowing agent, and identified with the formal set of mathematical theorems it knows in some language. Again, this isn't to say that's ALL the AGI is---that would be absurd. But it is one facet of the AGI. Whatever the heck the AGI is, it presumably knows some set of mathematical theorems in any given language (possibly the empty set, possibly a non-consistent set, etc.) There are various ways of classifying and even quantifying sets of mathematical theorems in a given language. See the slides I linked above for some details on this particular cross section.
Turing first wrote of that test in 1950. I don't think text was the primary way computers were interacted with back than, so if anything, it might be more plausible that cause for coming up with that interface was precisely because it's something that human-like intelligence is well-suited for rather than the reverse.
The point of using the terminal is to double-blind the role of the Judge. They have to decide whether they are communicating with a person or a machine - doing it face to face would rather give the game away.
When people imagine what a Turing Test conversation would look like, they frequently underestimate the conversation. I find Dennet's example of an imaginary Turing Test from Consciousness Explained to be a good counterexample:
Judge: Did you hear about the Irishman who found a magic lamp? When he rubbed it a genie appeared and granted him three wishes. “I’ll have a pint of Guiness!” the Irishman replied and immediately it appeared. The Irishman eagerly set to sipping and then gulping, but the level of Guiness in the glass was always magically restored. After a while the genie became impatient. “Well, what about your second wish?” he asked. Replied the Irishman between gulps, “Oh well, I guess I’ll have another one of these.”
CHINESE ROOM: Very funny. No, I hadn’t heard it– but you know I find ethnic jokes in bad taste. I laughed in spite of myself, but really, I think you should find other topics for us to discuss.
J: Fair enough but I told you the joke because I want you to explain it to me.
CR: Boring! You should never explain jokes.
J: Nevertheless, this is my test question. Can you explain to me how and why the joke “works”?
CR: If you insist. You see, it depends on the assumption that the magically refilling glass will go on refilling forever, so the Irishman has all the stout he can ever drink. So he hardly has a reason for wanting a duplicate but he is so stupid (that’s the part I object to) or so besotted by the alcohol that he doesn’t recognize this, and so, unthinkingly endorsing his delight with his first wish come true, he asks for seconds. These background assumptions aren’t true, of course, but just part of the ambient lore of joke-telling, in which we suspend our disbelief in magic and so forth. By the way we could imagine a somewhat labored continuation in which the Irishman turned out to be “right” in his second wish after all, perhaps he’s planning to throw a big party and one glass won’t refill fast enough to satisfy all his thirsty guests (and it’s no use saving it up in advance– we all know how stale stout loses its taste). We tend not to think of such complications which is part of the explanation of why jokes work. Is that enough?
Dennett: “The fact is that any program that could actually hold up its end in the conversation depicted would have to be an extraordinary supple, sophisticated, and multilayered system, brimming with “world knowledge” and meta-knowledge and meta-meta-knowledge about its own responses, the likely responses of its interlocutor, and much, much more…. Maybe the billions of actions of all those highly structured parts produce genuine understanding in the system after all.”
I don't put much stock in that possibility. Physicalism essentially closes the door on p-zombies, no?
It's far more arrogant to assume you're the odd one out. There's a reason we never use that kind of thinking. The following reasoning is clearly absurd:
I think I'm immune from radiation sickness. I know we have no evidence that anyone in history has been immune, but I've never been exposed, so I'm going to assume I'm immune.
> Maybe consciousness is caused by a particular benevolent parasite living in your gut, and whoever lacks that parasite lacks consciosness.
You could go with that explanation, but it's far more complex than necessary, which is a good indicator that it's not true.
A physicist won't be able to disprove the suggestion that gravity will only exist for as long as the Invisible Pink Unicorn wills it to exist. So what?
We can be pretty confident that consciousness arises from the normal activity of the brain. We can temporarily shut it down with anaesthetics, and see the diminished activity in parts of the brain.
I didn't say you should assume that either. You shouldn't assume either one until you have some reason to do so.
>A physicist won't be able to disprove the suggestion that gravity will only exist for as long as the Invisible Pink Unicorn wills it to exist. So what?
If I had some way of knowing which people are conscious, and I observed that everyone currently living and everyone in the past was always conscious, then it would indeed be silly for me to suggest that tomorrow many non-conscious people will be born --- as silly as your Invisible Pink Unicorn example. But the fact is, I do NOT know that anyone other than myself is conscious. I assume they are for the sake of adjudicating ethical dilemmas, but I only actually have that empirical data for myself.
>We can temporarily shut [consciousness] down with anaesthetics, and see the diminished activity in parts of the brain.
Here you seem to be assuming consciousness is identified with certain brain activities, but that's begging the question. For example, that would rule out computer simulations from being conscious, since you couldn't detect said brain activity in said computer. But it's far from settled whether or not consciousness is impossible in a computer simulation, in fact, many people think it IS possible. So unless you know something a lot of experts don't know, there isn't enough evidence to equate consciousness with certain brain activity.
Obviously (1) is possible, because for any particular configuration of atoms, it is theoretically possible for someone to assemble atoms together in that configuration. I could randomly put atoms together and, by sheer luck, assemble a living, breathing Abraham Lincoln. But since it happened by random chance, I wouldn't know for sure that this was really a living, breathing Abraham Lincoln. I might observe it for awhile and notice that, golly, it sure does seem to be a living, breathing Abraham Lincoln. But I could never rule out the possibility that it only looks like that initially and that it'll eventually collapse or fall into some infinite loop or something.
Intentionally building a machine and KNOWING that it has such-and-such properties of a thinking machine, is much more interesting. Such KNOWING is, itself, an act of thinking, and so there arises an interplay between the thinking of the created machine, and the thinking of the creator. Then you can start asking questions about whether the created machine, being a thinker, could also itself intentionally create another thinking machine, and so on.
If you're tired of the endless non-productive debate about whether thinking machines are possible, and you'd like to go to the next level, and take steps toward actually quantifying things, I'd invite you to read some papers I've written on the subject. To lessen the learning curve, here are some very approachable slides, and if you like those slides, you can click through to one of the papers. https://semitrivial.github.io/MeasuringIntelligence2019.pdf