Re:"Paste this post into ChatGPT 4"
by dietr1ch on Monday January 15, 2024 @03:06PM
(#39005416)
I'd guess that the tokenizer is just different and handles this in a "better" way.
[ Parent
| on HN ]
No, in both tokenizers Unicode tag-block code points like these are converted into bytes (two tokens per character), which is a fallback for code points uncommon enough to not warrant a dedicated token.