AI21 Labs concludes largest Turing Test experiment to date

Re:AI21 Labs concludes largest Turing Test experiment to date by tdba on Wednesday May 31, 2023 @11:51AM (#36139754)

In summary, humans win the Turing test ~2/3 of the time against current SOTA LLMs. One of the more interesting tactics used was to target a weakness of the LLMs themselves:

> ... participants posed questions that required an awareness of the letters within words. For example, they might have asked their chat partner to spell a word backwards, to identify the third letter in a given word, to provide the word that begins with a specific letter, or to respond to a message like "?siht daer uoy naC", which can be incomprehensible for an AI model, but a human can easily understand...

[ Parent | on HN ]

Re:AI21 Labs concludes largest Turing Test experiment to date by tornato7 on Wednesday May 31, 2023 @01:16PM (#36140877)

This is a pretty astute tactic. Because AI models operate on tokens and not characters, it's pretty easy to confuse them when you go down to the character level. Even asking "how many letters in _" is tough for LLMs.

[ Parent | on HN ]

Re:AI21 Labs concludes largest Turing Test experiment to date by miohtama on Wednesday May 31, 2023 @04:44PM (#36144090)

So not yet intelligent enough to pass Turing test, which is the goal of the test. I would not be worried about AI doomsday any time soon.

[ Parent | on HN ]

Re:AI21 Labs concludes largest Turing Test experiment to date by TheObviousOne on Wednesday May 31, 2023 @01:49PM (#36141296)

Interesting tactics might going the other direction.

Asking to generate in a super human capabilities...

write a 65 pages of poem about X...

[ Parent | on HN ]

Re:AI21 Labs concludes largest Turing Test experiment to date by choeger on Wednesday May 31, 2023 @04:48PM (#36144136)

Or just questions about three or four wildly different fields of science, sports and culture you happen to have more than a layman's understanding. If the answers are somewhat plausible, it's probably a model. Or your life partner.

[ Parent | on HN ]

Re:AI21 Labs concludes largest Turing Test experiment to date by 8bitsrule on Thursday June 01, 2023 @12:13AM (#36147495)

Or ask two or three historical questions that very few humans would know anything about ... but a well-trained model would.

[ Parent | on HN ]

Re:AI21 Labs concludes largest Turing Test experiment to date by chasd00 on Wednesday May 31, 2023 @10:57PM (#36147122)

> "?siht daer uoy naC"

That took me a while to figure out and I’m a human… as far as a I know anyway.

[ Parent | on HN ]