Re:OpenAI Tokenizer
by dqbd on Sunday April 09, 2023 @05:21PM
(#35506594)
Author here, you are correct! The issue here is due to the fact that a single user-perceived character might span into multiple tokens. This should be fixed now.
[ Parent
| on HN ]
Hey. Thank you! However has the fix not been deployed yet? Still shows broken UTF-8.
> a single user-perceived character might span into multiple tokens
Is this the way it works as designed or is this a bug?