Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ...
by Isinlor on Wednesday January 01, 2025 @06:11PM
(#42570085)
Transformers are very bad at counting in one feed forward pass, you need to explicitly tell them to use a counter in autoregressive fashion like here:
https://chatgpt.com/share/6775cb37-4198-8007-82cb-e897220827...
For example, the output for "how many x's are there in xaaax" is 3.
https://chatgpt.com/share/677591fe-aa58-800e-9e7a-81870387be...