I just tried and it was able to play tic tac toe, reverse a string (the string was "hello world.i am new to this so please forgive me if i can't reverse a sentence")
Hangman sort of worked but it said every letter I picked was correct and appears to have constructed a word based on my guesses. Very strange behavior
I couldn't make it reverse that correctly even after prompting it five times to fix its mistakes.
Most commonly it writes: ".yks eruz a eht rednu gnipacse ,yawa depoh ooragnak eht ,ylkciuQ"
It also can't find the mistakes in there for the life of it.
Taking [21063, 306, 11, 262, 479, 648, 38049, 45230, 1497, 11, 25071, 739, 262, 35560, 495, 6766, 13] to expect it to output back [13, 88, 591, 13724, 4496, 304, 4352, 2266, 28803, 19967, 541, 330, 325, 837, 88, 6909, 390, 381, 1219, 267, 273, 4660, 461, 304, 4352, 837, 2645, 74, 979, 84, 48] is a difficult problem that it is not well suited for.
Hangman and stone-paper-scissors though are entirely unsuited to a language model, at least one with a chat interface like ChatGPT, because they both require it to be able to store a secret. ChatGPT has no ability to do this: each time it returns a response by evaluating the previous conversation.
You could build a system that COULD play those games via an LLM but you'd have to write extra code to do it.
> I am thinking of a five letter word. You guess the word letter-by-letter (in the style of the game hangman) as many times as you like, but may only guess incorrectly six times. Please begin guessing
> Sure! Let’s start with the first letter of the word you’re thinking of. What is it?
lol
but it did get the picture after that. It does seem to be very fixated on guessing vowels though, after it has already exhausted all of them.
> Let's play hangman. I have thought of a 8 letter word, you have to guess it one guess at a time. You get to draw an ascii-art hangman too. Then YOU guess with a letter and I'll tell you if you were right or not. I won't give you a category clue. Start by drawing the hangman in a code block and then guessing a letter.
The same prompt against 3.5 didn't work - it didn't seem to be guessing likely next letters, and it couldn't keep track of how many body parts it should have drawn on the diagram.
> Let's play hangman. I'm thinking of a word: _______
It made a variety of guesses generally in the form of
> Based on the pattern you provided, I will guess the letter "I". Is there an "I" in the word?
My response was either "No [letter]" or an updated state of the word: "_RE_IE_".
BTW it doesn't just decode it, it also encodes it quite happily - in real time, as it is producing the output, sometimes unprompted. I once had GPT-4, when asked to produce SVG, produce it in form of an <img src="data:data:image/svg+xml;base64,..."> - and when I copy-pasted and rendered it, it was a valid SVG file with shapes inside.
Similarly, why would you expect a thing trained on the context of text would be good at reversing strings? It's amazing it's as good as it is at these things because it doesn't really make sense that it could do these things unless they trained it on reversed strings to add diversity but it's hard to gain context from a string backwards.
ALSO: your transcript doxs you and is hard to tell where your messages end and GPT begins. Just a heads up in case you don't want your full name leaked to whoever reads this
With reversing strings unfortunately it's not likely to achieve that considering its tokenization.
Actually a transformer doesn't accesses 'strings', all it processes are tokens which are then mapped to vectors by whatever embedding is applied. I think it will be extremely difficult for a transformer to do any of these tasks correctly until a successor model is adopted.
I don't have much hope of any reasonably complex symbolic processing of anything that it was not trained on. Some of these tasks are easy for a human to perform with paper and pencil and a set of rules; of course a human may get confused, but for that you write programs. Write code is one of GPT's skills but It is not "that" good with code for problems that are not mere small modification of problems it was trained on.