Re:QwQ: Alibaba's O1 Like Reasoning LLM
by fennecfoxy on Thursday December 05, 2024 @07:11AM
(#42327383)
I mean that's less of a reasoning capability problem and more of an architectural problem as afaik it's to do with the way that words are broken down into tokens, strawberry becomes straw + berry, or something like st-raw-b-erry as per the tokeniser.
An LLM trying to get the number of letters will just be regurgitating for the most part because afaik it has no way to actually count letters. If the architecture was changed to allow for this (breaking certain words down into their letter tokens rather than whole word tokens) then it may help, but is it worth it?
An LLM trying to get the number of letters will just be regurgitating for the most part because afaik it has no way to actually count letters. If the architecture was changed to allow for this (breaking certain words down into their letter tokens rather than whole word tokens) then it may help, but is it worth it?