You Should Probably Pay Attention to Tokenizers

Re:You Should Probably Pay Attention to Tokenizers by imtringued on Thursday October 24, 2024 @03:25AM (#41932924)

No LLM struggles with two digit arithmetic. 100 digit addition is possible with the use of state of the art position encodings. Counting is not bottlenecked by arithmetic at all.

When you ask an LLM to count the number of "r" in the word Strawberry, the LLM will output a random number. If you ask it to separate the letters into S t r a w b e r r y, then each letter is tokenized independently and the attention mechanism is capable of performing the task.

What you are doing is essentially denying that the problem exists.

[ Parent | on HN ]

Re:You Should Probably Pay Attention to Tokenizers by ithkuil on Thursday October 24, 2024 @08:42AM (#41934929)

Gpt4-o correctly answered

"How many letters "r" are in the word Frurirpoprar"

And it didn't use a program execution (at least it didn't show the icon and the answer was very fast so it's unlikely it generated an executed a program to count)

[ Parent | on HN ]

Re:You Should Probably Pay Attention to Tokenizers by danielmarkbruce on Thursday October 24, 2024 @12:43PM (#41937222)

I wouldn't consider that a thing that's going to work generally. That word may tokenize to one per char and have seen relevant data, or it may be relatively close to some other word and it's seen data which gives the answer.

[ Parent | on HN ]