OK, I can partly explain the LLM chess weirdness now

Re:OK, I can partly explain the LLM chess weirdness now by fragmede on Thursday November 21, 2024 @03:36PM (#42208372)

The illegal moves are interesting as it goes to "understanding". In children learning to play chess, how often do they try and make illegal moves? When first learning the game I remember that I'd lose track of all the things going on at once and try to make illegal moves, but eventually the rules became second nature and I stopped trying to make illegal moves. With an ELO of 1800, I'd expect ChatGPT not to make any illegal moves.

[ Parent | on HN ]

Re:OK, I can partly explain the LLM chess weirdness now by griomnib on Thursday November 21, 2024 @02:41PM (#42207832)

Likewise with LLM you don’t know if it is truly in the “chess” branch of the statistical distribution or it is picking up something else entirely, like some arcane overlap of tokens.

So much of the training data (eg common crawl, pile, Reddit) is dogshit, so it generates reheated dogshit.

[ Parent | on HN ]

[dead]

Re:OK, I can partly explain the LLM chess weirdness now by rscho on Friday November 22, 2024 @12:44AM (#42211464)

Understanding a rules-based system (chess) means to be able to learn non-probabilistic rules (an abstraction over the concrete world). Humans are a mix of symbolic and probabilistic learning, allowing them to get a huge boost in performance by admitting rules. It doesn't mean a human will never make an illegal move, but it means a much smaller probability of illegal move based on less training data. Asymptotically, performance from humans and purely probabilistic systems converge. But that also means that in appropriate situations, humans are hugely more data-efficient.

[ Parent | on HN ]

Re:OK, I can partly explain the LLM chess weirdness now by david-gpu on Friday November 22, 2024 @05:49AM (#42212744)

> in appropriate situations, humans are hugely more data-efficient

After spending some years raising my children I gave up the notion that humans are data efficient. It takes a mind numbing amount of training to get them to learn the most basic skills.

[ Parent | on HN ]

Re:OK, I can partly explain the LLM chess weirdness now by rscho on Friday November 22, 2024 @07:57AM (#42213448)

You could compare childhood with the training phase of a model. Still think humans are not data-efficient ?

[ Parent | on HN ]

Re:OK, I can partly explain the LLM chess weirdness now by david-gpu on Friday November 22, 2024 @08:03AM (#42213493)

Yes, that is exactly the point I am making. It takes many repetitions (epochs) to teach them anything.

[ Parent | on HN ]

Re:OK, I can partly explain the LLM chess weirdness now by rscho on Friday November 22, 2024 @08:07AM (#42213523)

Compared to the amount of data needed to train an even remotely impressive 'AI' model , that is not even AGI and hallucinates on a regular basis ? On the contrary, it seems to me that humans and their children are hugely efficient.

[ Parent | on HN ]

Re:OK, I can partly explain the LLM chess weirdness now by david-gpu on Friday November 22, 2024 @01:09PM (#42216057)

> On the contrary, it seems to me that humans and their children are hugely efficient.

Does a child remotely know as much as ChatGPT? Is it able to reason remotely as well?

[ Parent | on HN ]

Re:OK, I can partly explain the LLM chess weirdness now by rscho on Friday November 22, 2024 @01:56PM (#42216439)

I'd say the kid knows more about the world than ChatGPT, yes. For starters, the kid has representations of concepts such as 'blue color' because eyes... ChatGPT can answer difficult questions for sure, but overall I'd say it's much more specialized and limited than a kid. However, I also think that's mostly comparing apples and oranges, and that one's judgement about that is very personal. So, in the end I don't know.

[ Parent | on HN ]

Re:OK, I can partly explain the LLM chess weirdness now by meroes on Friday November 22, 2024 @09:16PM (#42218765)

A baby learns to walk and talk in 1 year. Compared to the number of PHDs and compute training these models, the baby is so far ahead in efficiency I marvel way more at their pace.

[ Parent | on HN ]

Re:OK, I can partly explain the LLM chess weirdness now by chongli on Friday November 22, 2024 @02:18AM (#42211795)

Neither AlphaZero nor MuZero can learn the rules of chess from an empty chess board and a pile of pieces. There is no objective function so there’s nothing to train upon.

That would be like alien archaeologists of the future finding a chess board and some pieces in a capsule orbiting Mars after the total destruction of Earth and all recorded human thought. The archaeologists could invent their own games to play on the chess board but they’d have no way of ever knowing they were playing chess.

[ Parent | on HN ]

Re:OK, I can partly explain the LLM chess weirdness now by BurningFrog on Friday November 22, 2024 @11:26AM (#42215139)

AlphaZero was given the rules of the game, but it figured out how to beat everyone else all by itself!

[ Parent | on HN ]

Re:OK, I can partly explain the LLM chess weirdness now by rscho on Friday November 22, 2024 @12:18PM (#42215615)

All by itself, meaning playing against itself...

Interestingly, Bobby Fischer did it in the same way. Maybe AlphaZero also hates chess ? :-)

[ Parent | on HN ]

Re:OK, I can partly explain the LLM chess weirdness now by sixfiveotwo on Friday November 22, 2024 @03:08PM (#42216997)

I think the article briefly touch on that topic at some point:

> For one, gpt-3.5-turbo-instruct rarely suggests illegal moves, even in the late game. This requires “understanding” chess. If this doesn’t convince you, I encourage you to write a program that can take strings like 1. e4 d5 2. exd5 Qxd5 3. Nc3 and then say if the last move was legal.

However, I can't say if LLMs fall in the "statistical AI" category.

[ Parent | on HN ]