Re:Understanding GPT Tokenizers
by mike_hearn on Friday June 09, 2023 @06:55AM
(#36255985)
Thanks for the link. That paper seems a bit different though. They're asking the model to do reasoning by emitting serialized graphs using a custom declarative data format, which it struggles with of course because it hasn't seen any such format before. Then they switch to asking it to emit code and it does better. But what I was meaning was more that code training helps it reason and speak better even in English, where no code is being emitted at all.
[ Parent
| on HN ]
To be fair Codex was much better than GPT-3 on reasoning benchmarks like MMLU and the like. And people have kind of noticed the Code trained models to reason better. Don't know if a paper was published about that though.