News for hackers. Stuff that matters.
Re:Virgin Galactic successfully flies tourists to space for ... by TaylorAlexander on Thursday August 10, 2023 @12:35PM (#37078349)
I really don’t think our current crop of chat bots can pass a Turing test. https://news.ycombinator.com/item?id=37057043
Re:Virgin Galactic successfully flies tourists to space for ... by justrealist on Thursday August 10, 2023 @01:06PM (#37078763)
None of those examples are GPT-4
Re:Virgin Galactic successfully flies tourists to space for ... by TaylorAlexander on Thursday August 10, 2023 @03:12PM (#37080451)
I paid for GPT-4 and noticed some weird behavior that seemed reproducible, so you could test for those things to reveal if you’re talking to a person or a chatbot. Biggest thing is you can ask it for a story of someone’s childhood (you can’t ask it for its childhood) and it will provide a plausible sounding but not internally consistent story.
Re:Virgin Galactic successfully flies tourists to space for ... by 93po on Thursday August 10, 2023 @04:26PM (#37081302)
You can also just ask it what time it is, or what day it is, or how much time has elapsed since your last message. It doesn't know any of these.
Re:Virgin Galactic successfully flies tourists to space for ... by justrealist on Thursday August 10, 2023 @04:37PM (#37081437)
That's an incredibly stupid Turing test, and you know it.
Re:Virgin Galactic successfully flies tourists to space for ... by TaylorAlexander on Thursday August 10, 2023 @04:45PM (#37081544)
If the robot cannot answer that question it is simple and effective test. The whole point of a turing test is that you cannot distinguish between a person and a robot. If you can wait one minute and then ask it how long it has been since your last request this sounds like a simple and effective test.
Re:Virgin Galactic successfully flies tourists to space for ... by justrealist on Thursday August 10, 2023 @05:34PM (#37082086)
Do you seriously think it would be hard for OpenAI to give GPT-4 a timer? The point of a Turing test is a passive evaluation of how intelligent a bot is. It's not a test that OpenAI is trying to pass.
Re:Virgin Galactic successfully flies tourists to space for ... by TaylorAlexander on Thursday August 10, 2023 @07:18PM (#37082979)
You misunderstand me. To me, a Turing test is passed when the interviewer cannot distinguish between a robot and a human. If OpenAI were to add a special timer function to GPT-4, then that specific question would no longer work to tell the difference, but that specific question is just one example of many types of questions one can use to tell that GPT-4 is not a human. GPT-4 with a clock does not change the fact that there are many ways to tell it is not human.

> The point of a Turing test is a passive evaluation of how intelligent a bot is.

I don't know what "the point" of the test is (that is subjective) but I know how the test works. You create a system where a tester and a subject can pass messages to each other but cannot see each other (so a chat system is great for this) and then the tester tries to determine if the subject is a robot or a person. If the tester cannot tell if the subject is a human or a robot, the robot has passed the test. Many people have pointed out the test is not necessarily useful for evaluating AI! An AGI may be extremely intelligent and useful and still not pass the Turing test, but it would nonetheless be true that it did not pass. However a highly intelligent AGI would probably be good at pretending to be a human.

> It's not a test that OpenAI is trying to pass.

We are in agreement on this. They are not trying to pass this test and their current system can not pass it. This is why I say the current system cannot pass a Turing test. I am not talking about a theoretical future system!

Re:Virgin Galactic successfully flies tourists to space for ... by 93po on Tuesday August 15, 2023 @03:57PM (#37138582)
It’s not so stupid in the context of a LLM that is good at summarizing but bad at common sense.
Re:Virgin Galactic successfully flies tourists to space for ... by Workaccount2 on Thursday August 10, 2023 @02:02PM (#37079579)
Cutting edge bots fail because they are too perfect to be human.
Re:Virgin Galactic successfully flies tourists to space for ... by TaylorAlexander on Thursday August 10, 2023 @02:20PM (#37079788)
Actually I think they make weird choices sometimes that could be tested for systematically if someone knew which model it was and had some time with it before the Turing test took place.
Re:Virgin Galactic successfully flies tourists to space for ... by AbrahamParangi on Thursday August 10, 2023 @03:17PM (#37080505)
The main issues with gpt and the Turing test are not that it isn’t smart enough but rather that fine tuning for helpfulness and harmlessness has turned it into a Quokka.

Completely harmless and naive in its attitudes towards harmful or controversial things, and this is trivially detectable.

Re:Virgin Galactic successfully flies tourists to space for ... by dTal on Thursday August 10, 2023 @10:17PM (#37084330)
This is a bit like saying that none of our current crop of computers are Turing complete, because their tape isn't infinite.

We will never achieve a theoretically perfect Turing machine, or a program indistinguishable from a real human in every single respect including funny edge cases. But we can get close enough that it's meaningful to use the theoretical ideal in a colloquial sense that means "crosses the threshold in most contexts". GPT models can actually keep up a half-decent facade against a suspicious interlocutor for a while, something no program has ever done before.

Re:Virgin Galactic successfully flies tourists to space for ... by TaylorAlexander on Thursday August 10, 2023 @10:38PM (#37084447)
I agree the software is doing better than ever before! However it still has serious shortcomings which is an active area of research. An AI researcher familiar with these issues could probably make an accurate detection in under one minute, so I don’t think this is an “infinite tape” issue this is simply an issue of performance. They are pretty good and they will get better, but I believe a true AGI would be able to mimic a human to properly pass the test.
Re:Virgin Galactic successfully flies tourists to space for ... by dTal on Friday August 11, 2023 @07:46AM (#37087583)
That's entirely fair. It definitely feels like some sort of threshold has been crossed, and the only term we have for this performance axis is "the Turing test", but you're absolutely right that LLMs are very far from perfect. It feels more like the beginning of a journey rather than the end. We should probably get in the habit of treating Turing Test performance as more like a continuous scale rather than a pass/fail condition.