That doesn’t explain why it still provides two answers (getting the number of letters right despite tokenization) and also doesn’t notice that they are contradictory.
Forgive the anthropomorphising, but I find it very similar to how I'd handle someone forcing me to explain two contradictory claims coherently.
But no one was forcing it to do that, and it doesn’t seem it has any awareness that it could proceed differently.
Re:What ChatGPT Can't Do
by chatgptexpert on Sunday December 25, 2022 @02:12PM
(#34130145)
This doesn't seem like a good explaination. If you ask ChatGPT to separate each letter in a word with a comma, it will do it correctly. So it can "see" and do operations at the character level.
https://www.gwern.net/GPT-3#bpes
The reason ChatGPT struggles with tasks involving individual character counting is that it can't see individual characters. The word "prime" is a single token (https://beta.openai.com/tokenizer) to the eyes of ChatGPT, and 1 is not prime.