See how ridiculous that sounds?
CoT or <work> tokens are not 'cheating' - they allow the system to have (as an analogy) a sheet of paper to work out the problem, instead of relying on a single step to jump through thousands of steps of reasoning. It's trained on the algorithmic reasoning embedded within natural language (Transformers<=>GATs+PE⊂GNNs, which are dynamic programmers, and align well with algorithmic reasoning), so the steps are necessary to lay out in as much detail.
What does that possibly have to do with GPT-4 shitting the bed if you draw the logic puzzle out of a deck of cards instead of cherry-picking and hinting it?
I cheated, and used bc, the unix utility.
These three maneuvers (the correct answer, the agency shown by cheating, and the crystallized intelligence of unix) handily demonstrate both my own intelligence, and that of my species.
The LLMs aren't in the same category of being, yet. It's not clear they ever will be; while I think silicon is a viable substrate for intelligent agents, it still strikes me as doubtful they'll be any more descended from the current machine learning systems as they are from Symbolics Lisp.
It seems to me that the ability to reason ( though primitive it may still be) is an emergent property of sufficiently large differentiable hash/lookup tables (and if you think about the mechanism of reasoning, that seems... Reasonable).
According to "folks in the know", the scaling limits of GPT architecture seem to be nigh, but if it were a bit higher I wouldn't be surprised if LLMs would have made it.
You might get more use out of these tools if you learn how to use them and these natural language instructions are how you use these tools.
It’s like calling the reverse gear on your car a cheat code!
I know what both 3.5-turbo and 4 say on my account, but I think I might be in some cost-effective inference holdout group because it’s always so dumb.