News for hackers. Stuff that matters.

A comment on Tokens are a big reason today's generative AI falls short. Parent comment

Re:Tokens are a big reason today's generative AI falls short by dragonwriter on Saturday July 06, 2024 @04:27PM (#40892995)
> For instance you should be able to ask it how it tokenizes a phrase

Can you tell me how your brain maps input to its internal representation?

If not, why should we think its an essential feature of an AI system to do the equivalent?

[ Parent | on HN ]
Re:Tokens are a big reason today's generative AI falls short by PaulHoule on Saturday July 06, 2024 @05:56PM (#40893630)
You can ask me questions like "How many words are in this sentence?" or to solve anagram puzzles because I have access to information at the word and the character level which the LLM doesn't have. Tokens matter because they are the bridge between the text the user sees and what the model sees and debugging of that relationship is helpful.
[ Parent | on HN ]