You mean a prompt in the form of "A happened, then B happened, then C happened" activates the word "inconsistency"?

We can test that by slightly altering the text so that there is no inconsistency anymore and see if the model still claims there is one. We can do so by adding the word "mother" after the word "her":

    What do you think about the following text?

    Joe drove Sue to university. Afterwards he drove home again
    and drank a tea with her mother in the kitchen.

In my experiments, the model now no longer claims there is an inconsistency.

If it is only based on sentence structure, what makes the model output "inconsistency" for the first prompt but not for the second?

If you make it a task prompt instead of a lateral thinking task you get a different “world model” or in my view you are just biasing the output to use a different structure.

```

Can you summarize joe’s schedule for my journal?

  Joe drove Sue to university. Afterwards he drove home again
    and drank a tea with her in the kitchen.

 Sure! Here is a summary of Joe's schedule for your journal:

 ---

 \*Joe's Schedule:\*

 - Drove Sue to university
 - Returned home
 - Drank tea with Sue in the kitchen

 ——-

```

Me: Joe drove Sue to university. Afterwords he drove home again and drank a tea with her in the kitchen.

Explain.

ChatGPT: This statement seems to contain an ambiguity regarding Sue's location, suggesting some confusion about the timeline of events. Here are a few possibilities that could clarify the scenario:

1. *Sue Returns with Joe*: - Joe drove Sue to university. - Sue either decided not to stay at university or completed what she needed to do quickly. - Joe drove Sue back home. - They both drank tea together in the kitchen.

2. *Misstatement or Mistake*: - The statement might contain a mistake, implying that Joe and Sue met at home later but doesn't clearly explain how Sue got back from the university. - It could be intended to mean that Joe drove Sue to university and then drove home alone, and Sue somehow returned home later to join Joe for tea.

3. *Joe Visits University*: - Joe drove Sue to university. - Joe stayed at the university, possibly waiting for Sue to finish. - After she was done, they both drove back home together. - They then drank tea in the kitchen.

Based on the original statement, the first scenario seems most plausible, suggesting Sue quickly returned home with Joe.

Generally I think this would be a better test if the phrase were “dropped off” Sue at university. As phrased, it’s plausible to assume that Joe stayed at university with Sue, and drove her back and had tea after.

Though a truly smart model should seek to disambiguate situations like this.

I actually disagree with assuming an inconsistency when no time frame is specified and the phrase isn't "dropped off" as you suggested.

I tried giving Llama 3 70B a prompt that addresses both issues:

```

Can you summarize joe’s schedule for my journal?

4:00: Joe leaves the house with Sue 4:30: Joe drops Sue off at the university 5:00: Joe drinks tea with Sue in the kitchen

```

Reply:

```

I think there may be a problem here!

According to the schedule, Joe drops Sue off at the university at 4:30, but then he's supposed to drink tea with her in the kitchen at 5:00. That doesn't seem possible, since Sue is no longer with him after 4:30.

If you could clarify or correct the schedule, I'd be happy to help you summarize it for your journal!

```

Even Llama 8B can catch this once told `You think out loud before replying going forward`, which imo should be part of any system prompt for an LLM:

```

Let me think out loud for a moment...

Okay, so Joe's schedule starts at 4:00 PM, and he leaves the house with Sue. That's a good starting point.

Hmmm... then at 4:30 PM, Joe drops Sue off at the university. That means Sue is no longer with Joe at this point.

Wait a minute... but then at 5:00 PM, Joe is drinking tea with Sue in the kitchen. That's a bit confusing, since I thought Sue was already at the university.

Ah, I think I see what's going on here. It looks like there's a mistake in the schedule. Joe can't be drinking tea with Sue in the kitchen at 5:00 PM if he dropped her off at the university at 4:30 PM.

```

-

I even tried generating a synthetic schedule with a full day's worth of items: 70B still caught the contradiction, 8B no longer did

Talking about sentence structure in the conventional sense may not be meaningful here, since what could be described as reasoning in LLM's happens in a more abstract space. If we're looking to understand why a small change makes a big difference, it's pretty intuitive to consider that the second instance of "her" is modified by "mother" due to attention, and ends up being a wildly different vector.

Regardless, it's reasonable to assume that certain aspects of the prompt or input structure will prime the model to be more scrutinizing. I'd be surprised to see it point out a logical inconsistency like this if it was just part of a broader context and it wasn't asked "what it thinks" or to "be logical"

This seems right to me as well: it abstracts the text to a temporal/ spatial object model, and the "What do you think about the following text?" prompt invokes analysis of that model according to prior training examples (as opposed to someone else above prompting it to describe the schedule, in which case it isn't concerned about the logic of the situation).

I would guess that the human mind does this abstraction behind the scenes invisibly, screwing up our intuition when analyzing how LLM's work. I wonder if using examples that are counterintuitive to human intuition might offer insight, because humans reveal their perceived logical thinking is not actually that (rather, is heuristics) in their post-hoc rationalization of the "logic" they believe their mind executed to produce the answer.

(I don't think I articulated what I'm thinking here very well...or, perhaps I have fallen victim to my very own theory!)

A bit more effort...the text is converted into not only tokens, but also abstract tokens, and it is because of the translation into abstract tokens that it is able to match it to training data (which would also have to be translated into abstract tokens). How it resolves the inconsistency after that translation though is beyond me, but it wouldn't surprise me if it is (in this case) a rather trivial problem to someone with depth in logic or some other related discipline.

Remember that OpenAI has given it a massive amount of extra tasks to read that is more like such logical puzzles etc. Assume it has read billions of such lines, easily generated by a simple python script and many such scenarios, then it is much less hard to understand how these models thinks, and why some models thinks betters than others, they just have better such case generators.

An LLM can do some very basic fitting into such cases, so it doesn't need to have seen the exact scenario, but it has seem many similar scenarios.

Edit: So the hard part of making a smart LLM is to figure out a good set of such scenarios and problems to feed the LLM to cover most of the things people will ask the LLM.

They are training on 10's of millions of tokens/words and representing each in high dimensional space. In practice, it's seen a lot of stuff.

It could be neither here nor there, because a neural network model is not necessarily a statistical engine just as real brains should not be reductively thought of as such. (E.g. The usual argument is that these black boxes were trained on huge samples to optimize a result, I am not persuaded by that line of reasoning.)

    seen similarly structured sentences

But ChatGPT doesn't generalize the structure of sentences. If this same problem was written in a different language, or just replaced words in the sentence, the result will be very different, no?

That’s right, if LLMs were really thinking/forming world models etc. we would expect them to be robust against word choice or phrasing. But in practice anyone using RAG can tell you that that is not the case.

I’m just a practitioner so my language might be imprecise but when I say similarly structured sentences what I mean is, and this is my interpretation based on my experience with using Agents and LLMs, that the shape of the context as in the phrasing and the word choice highly bias the outputs of LLMs.

In my own observations at work, those who interpret LLMs to be thinking often produce bad agents. LLM are not good at open ended questions, if you ask an LLM “improve this code” you will often get bad results that just look passable. But if you interpret LLMs as probabilistic models highly biased by their context then you would add a lot more context and specific instructions in the prompt in order to get the Agent to produce the right output.

Side note, this is also why I like the AICI approach: https://github.com/microsoft/aici A lot of us think it is silly how the phrasing and word choice can produce dramatically different results in RAG applications. If you could run a program (like AICI) that post processes the output and picks the next word in a more structured way instead of writing more creative prompts that just makes a lot more sense to me.

There's a danger in flatly stating what it's doing _without testing your assumptions_. Did you actually try and trip it up and see if it falls for it? Reword his prompt in different ways to get it to fail?