Bonus points for mentioning Gorgias :)

Re: wouldn't society race past this question without hesitation [if someone got lucky and produced what appears to be strong AI by just randomly throwing atoms together]?

Yes, and in fact some people would argue that is precisely what has in fact happened, the random atom-throwing-together process being called "evolution". And as you point out, philosophers would continue to debate about "is it REALLY strong AI?", just as nowadays philosophers debate about whether humans are machines or not.

These hypothetical debates will always exist, nothing will ever change that, not even if someone randomly creates a true strong AI by sheer luck.

My point is that there's a whole other half of the field, which ISN'T doomed to perpetual fruitless philosophical debating. The question of whether intelligent machines can exist is one that is doomed to fruitless debates forever. But the question of whether we can build specific machines and know with mathematical certainty how quantitatively intelligent they are---that is a much more fruitful area where progress can be made, theorems can be proved, mathematics can be applied, etc.

Unfortunately there's no agreement that we can actually quantify intelligence in a useful way, and believing that "ability to reason" and intelligence are synonymous, never mind evidence of sentience, is clearly incorrect.

Whatever human intelligence is, it has multiple dimensions, and being "able to reason" - whatever you assume that means - is a tiny element of a bigger picture.

It's practically a universal law of AI that proponents of AI inevitably understand AI in terms of their own niche interests - whether those are chess, go, e-sports, music, or driving. Only a field like CS could conceive of a test where the ability to emulate a conversation by typing lines of text on a terminal might be considered an indicator of operational sentience.

In reality all but the absolutely dumbest humans can operate a body fairly effortlessly, feed themselves and keep themselves clean, express characteristic preferences and act on them with agency, read and respond to human expressions and emotional cues, communicate internal emotional and psychological states verbally and non-verbally, improvise solutions to simple problems, maintain a fairly consistent emotional and psychological outlook that nonetheless develops over time, and make occasional surprising and unexpected - but not inhumanly weird - statements, observations, and actions.

How much of that is considered "being able to reason"?

If there's a way to quantify ability in all of the above, I'm not aware of it.

As a whole, an AGI is so general that we'll probably never be able to quantify its intelligence level. But we can quantify the intelligence of various "cross sections" of an AGI. As an extremely simple example, whatever the heck an AGI is, presumably an AGI can play chess. Therefore, we could instruct an AGI to compete in chess games and we could compute its Elo rating. This wouldn't be a measure of the AGI as a whole, but it would be a measure of the AGI as a chess-player.

More generally, we can consider the AGI to be a reinforcement learning agent: not that the AGI reduces to nothing but a reinforcement learning agent (that would be ridiculous), but rather, whatever the heck an AGI is, an AGI would certainly be capable of performing in reinforcement learning environments. Various methods have been proposed to measure RL agent intelligence, ranging from more practical to more philosophical.

Again, an AGI can be considered as a knowing agent, and identified with the formal set of mathematical theorems it knows in some language. Again, this isn't to say that's ALL the AGI is---that would be absurd. But it is one facet of the AGI. Whatever the heck the AGI is, it presumably knows some set of mathematical theorems in any given language (possibly the empty set, possibly a non-consistent set, etc.) There are various ways of classifying and even quantifying sets of mathematical theorems in a given language. See the slides I linked above for some details on this particular cross section.

> Only a field like CS could conceive of a test where the ability to emulate a conversation by typing lines of text on a terminal might be considered an indicator of operational sentience

Turing first wrote of that test in 1950. I don't think text was the primary way computers were interacted with back than, so if anything, it might be more plausible that cause for coming up with that interface was precisely because it's something that human-like intelligence is well-suited for rather than the reverse.

> Only a field like CS could conceive of a test where the ability to emulate a conversation by typing lines of text on a terminal might be considered an indicator of operational sentience

The point of using the terminal is to double-blind the role of the Judge. They have to decide whether they are communicating with a person or a machine - doing it face to face would rather give the game away.

When people imagine what a Turing Test conversation would look like, they frequently underestimate the conversation. I find Dennet's example of an imaginary Turing Test from Consciousness Explained to be a good counterexample:

Judge: Did you hear about the Irishman who found a magic lamp? When he rubbed it a genie appeared and granted him three wishes. “I’ll have a pint of Guiness!” the Irishman replied and immediately it appeared. The Irishman eagerly set to sipping and then gulping, but the level of Guiness in the glass was always magically restored. After a while the genie became impatient. “Well, what about your second wish?” he asked. Replied the Irishman between gulps, “Oh well, I guess I’ll have another one of these.”

CHINESE ROOM: Very funny. No, I hadn’t heard it– but you know I find ethnic jokes in bad taste. I laughed in spite of myself, but really, I think you should find other topics for us to discuss.

J: Fair enough but I told you the joke because I want you to explain it to me.

CR: Boring! You should never explain jokes.

J: Nevertheless, this is my test question. Can you explain to me how and why the joke “works”?

CR: If you insist. You see, it depends on the assumption that the magically refilling glass will go on refilling forever, so the Irishman has all the stout he can ever drink. So he hardly has a reason for wanting a duplicate but he is so stupid (that’s the part I object to) or so besotted by the alcohol that he doesn’t recognize this, and so, unthinkingly endorsing his delight with his first wish come true, he asks for seconds. These background assumptions aren’t true, of course, but just part of the ambient lore of joke-telling, in which we suspend our disbelief in magic and so forth. By the way we could imagine a somewhat labored continuation in which the Irishman turned out to be “right” in his second wish after all, perhaps he’s planning to throw a big party and one glass won’t refill fast enough to satisfy all his thirsty guests (and it’s no use saving it up in advance– we all know how stale stout loses its taste). We tend not to think of such complications which is part of the explanation of why jokes work. Is that enough?

Dennett: “The fact is that any program that could actually hold up its end in the conversation depicted would have to be an extraordinary supple, sophisticated, and multilayered system, brimming with “world knowledge” and meta-knowledge and meta-meta-knowledge about its own responses, the likely responses of its interlocutor, and much, much more…. Maybe the billions of actions of all those highly structured parts produce genuine understanding in the system after all.”