News for hackers. Stuff that matters.
Re:"Paste this post into ChatGPT 4" by goodside on Monday January 15, 2024 @03:31PM (#39005692)
No, in both tokenizers Unicode tag-block code points like these are converted into bytes (two tokens per character), which is a fallback for code points uncommon enough to not warrant a dedicated token.

There are no comments on this comment yet.