Re:Claude Token Counter, now with model comparisons
by yorwba on Monday April 20, 2026 @05:23AM
(#47831976)
Anthropic was already special-casing case-folding in their tokenizers before this recent change: https://transformer-circuits.pub/2025/attribution-graphs/met... "The tokenizer the model was trained with uses a special “Caps Lock” token" (⇪). Their visualizations for Claude 3.5 Haiku also show the Title Case token (↑).
This is similar to what the TokenMonster tokenizer does: https://github.com/alasdairforsythe/tokenmonster