Do Large Language Models learn world models or just surface statistics?

Re:Do Large Language Models learn world models or just surfa... by bsk7 on Sunday January 22, 2023 @11:50AM (#34479266)

> We only need a single counter example to show that Othello-GPT does not have a systemic understanding of the rules, only a statistical inference of them.

I don't agree with this binary interpretation. This only indicates that any systemic understanding the model has internally built is not complete. We are not trying to assess completeness of that understanding. E.g. if you had a traditional rules engine for Othello, and you removed one rule that could result in illegal moves, does that make that rule engine a statistical model all of a sudden?

[ Parent | on HN ]

Re:Do Large Language Models learn world models or just surfa... by nickelpro on Sunday January 22, 2023 @12:12PM (#34479569)

No, it makes it an engine for a game that is not Othello.

An inaccurate engine and a statistical one are indistinguishable. And the authors provide no evidence to prefer the former.

Nothing is ruled out, of course, but there's not strong or compelling evidence either.

[ Parent | on HN ]

Re:Do Large Language Models learn world models or just surfa... by bsk7 on Sunday January 22, 2023 @12:24PM (#34479739)

> An inaccurate engine and a statistical one are indistinguishable.

How so? The paper is trying to assess if some semantic understanding is being created by the underlying model, not if a completely accurate one is. If such a world model maps to Othello-prime (gleaning concepts like tiles, colors etc), that is still a very interesting result.

[ Parent | on HN ]