News for hackers. Stuff that matters.

A comment on Claude Token Counter, now with model comparisons. Parent comment

Re:Claude Token Counter, now with model comparisons by dannyw on Monday April 20, 2026 @01:54AM (#47830832)
LLMs are explicitly designed to handle, and also possibly 'learn' from different tokens encoding similar information. I found this video from 3blue1brown very informative: https://www.youtube.com/watch?v=wjZofJX0v4M

Also, think about how a LLM would handle different languages.

[ Parent | on HN ]

There are no comments on this comment yet.