Re:OK, I can partly explain the LLM chess weirdness now
by jeremyjh on Friday November 22, 2024 @09:21AM
(#42214027)
Yes, I don't even know what it means to say its 1800 strength and yet plays illegal moves frequently enough that you have to code retry logic into the test harness. Under FIDE rules after two illegal moves the game is declared lost by the player making that move. If this rule were followed, I'm wondering what its rating would be.
[ Parent
| on HN ]
Re:OK, I can partly explain the LLM chess weirdness now
by og_kalu on Friday November 22, 2024 @11:43AM
(#42215285)
>Yes, I don't even know what it means to say its 1800 strength and yet plays illegal moves frequently enough that you have to code retry logic into the test harness.
People are really misunderstanding things here. The one model that can actually play at lichess 1800 Elo does not need any of those and will play thousands of moves before a single illegal one. But he isn't just testing that one specific model. He is testing several models, some of which cannot reliably output legal moves (and as such, this logic is required)