The Geometry of Truth: Do LLM's Know True and False

Re:The Geometry of Truth: Do LLM's Know True and False by alphablended on Friday October 20, 2023 @04:29AM (#37953597)

> An example I can remember from the paper is "Who is Tom Cruise's mother? [Tom Cruise's mother's name]" paired with "Who is [Tom Cruise's mother's name]'s son? [incorrect answer or "Can't answer that"]".

The paper is, apparently, still under review.

In the mean time, may I suggest you to verify that example by yourself?

[ Parent | on HN ]

Re:The Geometry of Truth: Do LLM's Know True and False by kybernetikos on Friday October 20, 2023 @04:48AM (#37953730)

I'm seeing exactly what lucubratory described with ChatGPT. If the information is in the context window, it has no trouble working out the reverse.

    Me: Who is tom cruises mother?
    ChatGPT: Tom Cruise's mother is Mary Lee Pfeiffer.
    User: Who is Mary Lee pfeiffers son
    ChatGPT: Mary Lee Pfeiffer's son is the famous actor, Tom Cruise.

But if you ask my second question directly into a fresh session, it doesn't know the answer.

Interestingly though, you can give it additional clues and it'll get it. https://chat.openai.com/share/893c1088-6718-4113-a3f1-cf273d...

[ Parent | on HN ]

Re:The Geometry of Truth: Do LLM's Know True and False by alphablended on Saturday October 21, 2023 @04:00AM (#37964962)

I misread lucubratory's comment: indeed I see the same as you do. I only tried asking both question in the same session. I didn't see that point in the paper when I quickly skimmed through it to find the relevant part.

I also agree with him about humans capable of the same "errors".

[ Parent | on HN ]

Re:The Geometry of Truth: Do LLM's Know True and False by lucubratory on Saturday October 21, 2023 @07:46PM (#37971548)

Her, but thank you for rereading the comment, I appreciate it.

[ Parent | on HN ]