News for hackers. Stuff that matters.
Re:What Is ChatGPT Doing and Why Does It Work? by BoorishBears on Tuesday June 18, 2024 @05:19PM (#40722274)
I actually disagree with assuming an inconsistency when no time frame is specified and the phrase isn't "dropped off" as you suggested.

I tried giving Llama 3 70B a prompt that addresses both issues:

```

Can you summarize joe’s schedule for my journal?

4:00: Joe leaves the house with Sue 4:30: Joe drops Sue off at the university 5:00: Joe drinks tea with Sue in the kitchen

```

Reply:

```

I think there may be a problem here!

According to the schedule, Joe drops Sue off at the university at 4:30, but then he's supposed to drink tea with her in the kitchen at 5:00. That doesn't seem possible, since Sue is no longer with him after 4:30.

If you could clarify or correct the schedule, I'd be happy to help you summarize it for your journal!

```

Even Llama 8B can catch this once told `You think out loud before replying going forward`, which imo should be part of any system prompt for an LLM:

```

Let me think out loud for a moment...

Okay, so Joe's schedule starts at 4:00 PM, and he leaves the house with Sue. That's a good starting point.

Hmmm... then at 4:30 PM, Joe drops Sue off at the university. That means Sue is no longer with Joe at this point.

Wait a minute... but then at 5:00 PM, Joe is drinking tea with Sue in the kitchen. That's a bit confusing, since I thought Sue was already at the university.

Ah, I think I see what's going on here. It looks like there's a mistake in the schedule. Joe can't be drinking tea with Sue in the kitchen at 5:00 PM if he dropped her off at the university at 4:30 PM.

```

-

I even tried generating a synthetic schedule with a full day's worth of items: 70B still caught the contradiction, 8B no longer did

There are no comments on this comment yet.