News for hackers. Stuff that matters.
Re:Claude Token Counter, now with model comparisons by yorwba on Monday April 20, 2026 @05:23AM (#47831976)
Anthropic was already special-casing case-folding in their tokenizers before this recent change: https://transformer-circuits.pub/2025/attribution-graphs/met... "The tokenizer the model was trained with uses a special “Caps Lock” token" (⇪). Their visualizations for Claude 3.5 Haiku also show the Title Case token (↑).

This is similar to what the TokenMonster tokenizer does: https://github.com/alasdairforsythe/tokenmonster

There are no comments on this comment yet.