I played it and made the wrong move and it still said I won.

See if you can get it to play better within your particular context even. I couldn't get it to improve!

I think switching to GPT-4 I managed to get it to learn by playing itself, and helping out and explaining when that went wrong. I did hit the usage cap though.

Would not recommend trying to teach it Global Thermonuclear War at this time.

one was in the training set and one wasn't

It stands to reason that tic tac toe would be in the training set, but a random stack of objects wouldn't.

How many people play text tic-tic-toe on the internet? Puzzle questions would seem more common.

That’s interesting. I just played a game with ChatGPT4 and it went smoothly. It gave me the rules, managed the game board, and let me know when I made an invalid move.

Some games go smoothly, some don't (occasionally it plays impossible moves or plays after I've already won, although not often). Even when games go smoothly, it seems to play very poorly even when told to win and even when I told it to explain the optimal tic-tac-toe strategy in advance.

While it is pretty impressive that it can play at all, it does make me think its intelligence is a bit less generalized and/or "human" than some people think. I think any human that can play chess to the level GPT4 can (apparently pretty high) would easily be able to figure out tic-tac-toe, even without our equivalent of "training data."