News for hackers. Stuff that matters.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by sealeck on Wednesday January 01, 2025 @02:06PM (#42568325)
It also fails on things that aren't actual words

For example, the output for "how many x's are there in xaaax" is 3.

https://chatgpt.com/share/677591fe-aa58-800e-9e7a-81870387be...

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by Isinlor on Wednesday January 01, 2025 @06:11PM (#42570085)
Transformers are very bad at counting in one feed forward pass, you need to explicitly tell them to use a counter in autoregressive fashion like here:

https://chatgpt.com/share/6775cb37-4198-8007-82cb-e897220827...