That is a poor summary.

TL;DR: actually, they might. If I'm reading the visualization correctly, and not just projecting my own prior expectations, it seems to imply that a distinct and general concept of "true" and "false" (or at least "known true" and "known false") emerges in the latent space.

It'll be interesting to explore how robust it is for statements crafted so that they surely did not occur in the training data, but their truth value could be indirectly inferred from information in the training data.

I can already answer you that question based on the "reversal curse" paper. LLMs trained on “A is B” fail to learn “B is A”. It will only work if you explicitly generate text connecting B to A before training.

More generally, the organic training data is almost always incomplete. Samples of text leave many assumptions and inferences hidden. Like a math problem, you see the statement, but don't know the answer until you work it out. Or like a puzzle, you see the pieces but don't know the big picture until you fit those pieces together.

That is how training text samples are like unsolved enigmas, we train our models on undigested text. Often the pieces are spread over many training examples that almost aways appear separately, never together have the chance to draw a conclusion from them. Search is needed, augmenting training examples with supporting data.

Neural nets are smart at inference time but dumb at training time. They don't make those connections when they train. Instead, we need to draw those connections out by generating new text. We need to benefit from inference-time smarts before training. That means we need to use current LLMs to write the dataset of next LLMs.

All the best LLMs today used a big piece of synthetic data, including GPT-4. Datasets like Orca, Phi-1.5, ShareGPT, etc. It's also the best way to create small models (<10B) that actually work, you need very high quality, high diversity synthetic data.

I wonder how well it would work if llms would do "online learning". After every interaction they'd be asked to summarise new knowledge (assuming the input is trusted) and create multiple variations on it, then train on it. I wonder if such a model would improve or degrade over time.

I don't get it. A is B doesn't always mean B is A.

Could you give an example?

Cows are mammals, but mammals are not cows.

"Cows are mammals" means: Array($mammals).includes($cows).

It does not mean $cows === $mammals.

$a === $b ("A is B") implies that $b === $a (B is A), in all cases that I know of.

What cases are there for $a === $b? I guess there's synonyms and rephrasing, but in general it's not a useful relationship in language

Batman is Bruce Wayne.

From this paper:

https://owainevans.github.io/reversal_curse.pdf

"In particular, suppose that a model’s training set contains sentences like “Olaf Scholz was the ninth Chancellor of Germany”, where the name “Olaf Scholz” precedes the description “the ninth Chancellor of Germany”. Then the model may learn to answer correctly to “Who was Olaf Scholz? [A: The ninth Chancellor of Germany]”. But it will fail to answer “Who was the ninth Chancellor of Germany?” and any other prompts where the description precedes the name."

The operator are, which happens to look similar to an english word, has a meaning that depends on context. Another example is 5/2 means a simple fraction in one context, or a class of polytopes in another.

GP meant that LLMs trained to learned "A is a B" won't necessarily be able to answer "give an example of B". Eg given the fact "Foo is the capital city of Bar", it might be able to complete "Foo is the capital city of _" but not "The capital city of Bar is _".

(Which isn't necessarily a evidence against truth models for out-of-distribution facts, it could just be a matter of indexing.)

Yes it can only learn to solve the problems that it was trained to solve. Not surprising really. Tautological.

Right, but that ignores the key piece, which is whether such systems can infer the equivalence relation denoted by "is". A is A. A is B implies B is A. A is B and B is C implies A is C. When a system sees that A is B, yet cannot infer that B is A, it exhibits an asymmetry, which is interesting. Whether this asymmetry exists in the underlying language is unclear.

> When a system sees that A is B, yet cannot infer that B is A

So the issue is that they cannot infer that general rule due to a fundamental limitation of the transformer LLM architecture, not just a training data issue? I skimmed the paper and it seems to be the case.

> A is B implies B is A.

I get the overall idea, but this statement isn't always true, right?

Correct. Not in natural language. These are natural language systems, not logic systems.

"The sky is blue" does not imply "blue is the sky".

There's an asymmetry here, too. "The sky is ${color}", with no extra context, has one obvious answer for us living here, today. Whereas for "Blue is the color of ${thing}", with no extra answer, there's an insane amount of equally sensible substitutions for ${thing}. Without extra content, the model has no reason to privilege "sky" over any other equally valid answer.

>"Eg given the fact "Foo is the capital city of Bar", it might be able to complete "Foo is the capital city of _" but not "The capital city of Bar is _"."

Is that really so? It would seem to be well within what I thought they were capable of based on all the other things they can do correctly.

The reversal curse itself is real. An example I can remember from the paper is "Who is Tom Cruise's mother? [Tom Cruise's mother's name]" paired with "Who is [Tom Cruise's mother's name]'s son? [incorrect answer or "Can't answer that"]". It is an interesting side effect of the way they think and a significant annoyance in getting them to work as we want them to.

That said, I think people make too much of it as an "LLMs can't reason" point, when I don't think that's accurate. What it says is that LLMs instant recall is not logically bidirectional, but this is something that humans do as well. Humans take longer to respond to (and are less accurate answering) "Who is Tom Cruise's mother?" than "Who is [her name]'s son?". At least for me, when I get questions that are the "wrong way around", I have to literally run through it logically in my head, generally along the lines of "(What does that name remind me of, is she a spy? Or her son is a spy? Is she a fictional character? Wait I think this is is a celebrity thing, which spy celebrity has her as his mum? Oh yeah, Tom Cruise.) [Out loud:] Tom Cruise."

Also, some people misunderstand the actual deficiency, and think that the LLM can't answer the question at all, rather than just zero shot. The LLM can answer the question if it has the information in context, it can reason "If A=B, then B=A" just fine. It just can't do the less popular halves of AB equivalencies zero shot.

> An example I can remember from the paper is "Who is Tom Cruise's mother? [Tom Cruise's mother's name]" paired with "Who is [Tom Cruise's mother's name]'s son? [incorrect answer or "Can't answer that"]".

The paper is, apparently, still under review.

In the mean time, may I suggest you to verify that example by yourself?

I'm seeing exactly what lucubratory described with ChatGPT. If the information is in the context window, it has no trouble working out the reverse.

    Me: Who is tom cruises mother?
    ChatGPT: Tom Cruise's mother is Mary Lee Pfeiffer.
    User: Who is Mary Lee pfeiffers son
    ChatGPT: Mary Lee Pfeiffer's son is the famous actor, Tom Cruise.

But if you ask my second question directly into a fresh session, it doesn't know the answer.

Interestingly though, you can give it additional clues and it'll get it. https://chat.openai.com/share/893c1088-6718-4113-a3f1-cf273d...

I misread lucubratory's comment: indeed I see the same as you do. I only tried asking both question in the same session. I didn't see that point in the paper when I quickly skimmed through it to find the relevant part.

I also agree with him about humans capable of the same "errors".

Her, but thank you for rereading the comment, I appreciate it.

Good point. In some ways this kind of error makes it feel like its process is very comparable to what humans do.

Yeah, I'm counting this as yet another weak but positive evidence that we've hit on something fundamental here - if not universal, then at least to the path evolution took that led to a human mind.

Yeah, I agree. There is exactly one non-LLM entity in existence that can reason this generally and this well in human languages, and that's us, human beings. We built LLMs by taking inspiration from the human brain and trying to approximate it with neural networks that were often able to achieve intelligent-ish performance on tasks in narrow domains, and eventually we stumbled on an architecture that is truly general, even if it's generally dumb. It would be an absurd coincidence to me if that architecture, LLMs, actually had nothing in common with how humans think. None of that means it is the best architecture for thinking like humans, or that we just need to scale it up to get to super-intelligence, or that it is currently as smart as a human being. But it just doesn't seem plausible that it behaves so much like a human mind if there's really nothing in common underneath.

If I say to chat gpt

Adrian Tompkins was the ninth mayor of the town of Wolverhillington. Who was the ninth mayor of Wolverhillington?

Then it correctly responds

The ninth mayor of the town of Wolverhillington was Adrian Tompkins.

What am I doing wrong?

Nothing. It's not an inference problem but a training one. If you train on the sequence,

"Adrian Tompkins was the ninth mayor of the town of Wolverhillington"

And later ask in inference,

"Who was the ninth mayor of Wolverhillington?",

It might not return the answer.

Literally from the abstract of the accompanying paper:

"Overall, we present evidence that language models linearly represent the truth or falsehood of factual statements."

You didn't even read anything before commenting just off the headline, did you?

> You didn't even read anything before commenting just off the headline, did you?

Welcome to hacker news. :)

I think people fixate on the word "knows" too much. They can clearly identify the difference between true and false statements, in the sense that if you provide them some statement it can identify it as true or false with some large degree of accuracy that is far more than you'd expect from random chance. This takes like 5 seconds to verify with chatgpt and you can see it as a feature in these data sets.

Whether it "knows" true and false or _anything at all_ is a completely different question.