https://chatgpt.com/share/79c5c6e1-e6a9-441b-acb3-54882303a8...
Of course as usual, LLMs are horrible with Math.
Funny enough, the next time it verified the word count by counting it out until I specifically told it to use Python
https://chatgpt.com/share/79e7b922-9b0f-4df9-98d0-2cd72d7041...
Even still, if you ask it for token count it would still fail. My point is that it can’t count, the circuitry required to do so seems absent in these models
A child will start to speak at around the age of one, but most will be about two before they start to count. And it is even longer (maybe the age of three to four) before they understand cardinality and can reliably follow “simple” instructions like “bring me four blocks”.
And basic arithmetic without counting on their fingers is usually not picked up until they are around six or seven.
All the LLMs and multi-modal models I've seen lack concrete reasoning. For instance, ask ChatGPT to perform 2 tasks, to summarize a chunk of text and to count how many words are in this chunk. ChatGPT will do a very good job summarizing the text and an awful job at counting the words. ChatGPT and all the transformer based models I've seen fail at similar concrete/mathematical reasoning tasks. This is the core problem of creating AGI and it generally seems like no one has made any progress towards synthesizing something with both a high and low level of intelligence.
My (unproven and probably incorrect) theory is that under the hood these networks lack information processing loops which make recursive tasks, like solving a math problem, very difficult.