Re:Tokens are a big reason today's generative AI falls short
by PaulHoule on Saturday July 06, 2024 @05:56PM
(#40893630)
You can ask me questions like "How many words are in this sentence?" or to solve anagram puzzles because I have access to information at the word and the character level which the LLM doesn't have. Tokens matter because they are the bridge between the text the user sees and what the model sees and debugging of that relationship is helpful.
[ Parent
| on HN ]
There are no comments on this comment yet.