Tracing the thoughts of a large language model

Re:Tracing the thoughts of a large language model by born1989 on Thursday March 27, 2025 @03:03PM (#43496901)

Thanks! Isn’t “an Astronomer” a single word for the purpose of answering that question?

Following your comment, I asked “Give me pairs of synonyms where the last letter in the first is the first letter of the second”

Claude 3.7 failed miserably. Chat GPT 4o was much better but not good

[ Parent | on HN ]

Re:Tracing the thoughts of a large language model by nearbuy on Thursday March 27, 2025 @03:11PM (#43496993)

Don't know about Claude, but at least with ChatGPT's tokenizer, it's 3 "words" (An| astronom|er).

[ Parent | on HN ]

Re:Tracing the thoughts of a large language model by colah3 on Thursday March 27, 2025 @03:13PM (#43497016)

"An astronomer" is two tokens, which is the relevant concern when people worry about this.

[ Parent | on HN ]

Re:Tracing the thoughts of a large language model by philomath_mn on Thursday March 27, 2025 @04:41PM (#43497925)

That is a sub-token task, something I'd expect current models to struggle with given how they view the world in word / word fragment tokens rather than single characters.

[ Parent | on HN ]