OK, but the LLM is still playing without a board to look at, except what's "in its head". How often would 1800 ELO chess players make illegal moves when playing only using chess notation over chat, with no board to look at?

What might be interesting is to see if there was some sort of prompt the LLM could use to help itself; e.g., "After repeating the entire game up until this point, describe relevant strategic and tactical aspects of the current board state, and then choose a move."

Another thing that's interesting is the 1800 ELO cut-off of the training data. If the cut-off were 2000, or 2200, would that improve the results?

Or, if you included training data but labeled with the player's ELO, could you request play at a specific ELO? Being able to play against a 1400 ELO computer that made the kind of mistakes a 1400 ELO human would make would be amazing.

MaiaChess [1] supposedly plays at a specific ELO, making similar mistakes a human would make at those levels.

It looks like they have 3 public bots on lichess.org: 1100, 1500, and 1900

[1] https://www.maiachess.com/

You can make it available to the player and I suspect it wouldn’t change the outcomes.

The LLM can't refer to notes, it is just relying on its memory of what input tokens it had.

Sure but I'm better than 99% of people at chess and if I was playing under those conditions there is a high chance I would make an illegal move.