News for hackers. Stuff that matters.
Re:You Should Probably Pay Attention to Tokenizers by ithkuil on Thursday October 24, 2024 @08:42AM (#41934929)
Gpt4-o correctly answered

"How many letters "r" are in the word Frurirpoprar"

And it didn't use a program execution (at least it didn't show the icon and the answer was very fast so it's unlikely it generated an executed a program to count)

Re:You Should Probably Pay Attention to Tokenizers by danielmarkbruce on Thursday October 24, 2024 @12:43PM (#41937222)
I wouldn't consider that a thing that's going to work generally. That word may tokenize to one per char and have seen relevant data, or it may be relatively close to some other word and it's seen data which gives the answer.