Chess doesn't require understanding human goals if you're an AI, just running through the space of winning moves that your training encountered. An AI doesn't think "ah, rook to E4, knowing Kasparov, he will then castle and I can do X". It'll just go "lol I encountered 3 486 864 games with this particular board pattern and 49 652 lead to a victory lemme pick one for the lulz".

Chess is really just about searching through as much of the space as possible, and you can take a few seconds. And believe me, there's a whole lot more stupid behaviours you can encounter on the road while you have to react in less than a second.

> It'll just go "lol I encountered 3 486 864 games with this particular board pattern and 49 652 lead to a victory lemme pick one for the lulz".

That isn't really an accurate description of how the modern DL bots work. They don't need to reference any database of past games while they play.

Chess AIs don't need to develop an understanding of the goal, they're hard-wired with the rules of the game and its goal-states.

As an experiment, try making a chess AI without explicitly giving it the rules of the game and see how it performs.

I think that's already been done: https://en.wikipedia.org/wiki/MuZero

Almost, but not quite, exactly unlike chess.

So not like chess, but like how chess was supposed to be?

Chess is a perfect information game. Both opponents can see the entire board at all times. Reality, nor even driving, is not anywhere close to that. Weather, time of day, climate, seasonality, and other drivers throw constant and voluminous amounts of unpredictability into the mix.

> Reality, nor even driving, is not anywhere close to that.

This reminds me of the kinds of intuitions people had about Chess and Go. In retrospect they seem silly, but it made plenty of sense to them at the time. The fact was that there was a solution that machines could use that humans couldn't use. Naturally, a solution that humans couldn't use was hard for them to anticipate being effective.