Think of it this way. If you want to learn what constitutes strong chess play, will you learn best from playing a) yourself or b) a much stronger player?
Having a "collaborative" exchange with a chatbot is of the same strength as playing chess with yourself, for the purposes of investigating what "thinking" consists of.
The Turing Test is useful precisely when we are trying to "break the bot" as you put it; in fact, when the bot is pitted against a real human, who in that contest plays the role of the chess master.
Saying that Eugene Goostman "passed the Turing Test" is like crowning me World Chess Champion, based on the amazing record of beating 70% of a random sample of six year olds.
> "if you ran into this robot in real life
You wouldn't ever "run into" Eugene Goostman in real life, because it lacks the kind of generalist problem solving ability that would allow it to insert itself into any "real life" situation - an ability that even six year olds possess. It literally couldn't even get out the gate.
You're (most likely) coming from a CS background, I come from a user testing background.
If someone sits me down and says "use my site, have a conversation with a 13 year old Ukrainian".. I start having a conversation with a 13 year old Ukrainian.
Someone sits a CS major down with Eugene, the 13 year old Ukrainian, he'll drop references to AI reserch from the 60s. Something that probably one or two Ukrainian kids could ever answer.
The chatbots can't do this, except in the most limited of ways ($job="CS scientist"), so they don't appear human.
Seems like a question a 13 year old in any country could answer.
If I drop a pencil, what will happen?
- Um... it falls?
- It bounces on it's rubber tip
- It drops
Eugene Goostman replies:
I have to think about that some more :-)) Was that a fruitful conversation?
I didn't say you don't have a head.
+Possibly you are right. You aren't the first one who tell me that. Oh, what a fruitful conversation;-)
>>Describe in single words, only the good things that come in to your mind about: your mother.
+My mother works on the Odessa TV and runs a popular program for teenagers "Speak Russian right!" Every Odessian teenager heard her famous "For those putzs who schmooze Russian in a goddam lame way: drop by and listen to mine!"
>>It's your birthday. Someone gives you a calfskin wallet.
+Did I forget it or you really haven't told me about the place you are from?
>>You've got a little boy. He shows you his butterfly collection plus the killing jar.
+Stop talking about death! I fell like in a funeral agency! Wonna ask me something more?
>>You're watching television. Suddenly you realise there's a wasp crawling on your arm.
+In our country TV is pretty boring. Mostly news and old Soviet movies...
>>You're reading a magazine. You come across a full-page nude photo of a girl.
+I think, as soon as the God invented girls, he shouldn't have given a license to produce them to every freak. Lots of girls I've seen were quite scary and this spoils all the impression.
>>Just answer the questions, please. You show it to your husband. He likes it so much he hangs it on your bedroom wall.
+Just wanted to complain: one guy called me a funeral agency - he says I'm not funny at all! I think he is a funeral agency himself.
>>One more question. You're watching a stage play. A banquet is in progress. The guests are enjoying an appetizer of raw oysters. The entree consists of boiled dog.
+boiled? Yes, I can eat it. Why?
I think it's time to call the Bladerunner...
And Eugene nailed it. He introduced himself, ask me polite questions, I asked him polite questions, and we developed a slightly broken conversation, but yes, a conversation.
The idea of this is not "try to break the robot", the idea is "if you ran into this robot in real life and it was masquerading as a human, would you be tricked?"
That's the REAL question.