News for hackers. Stuff that matters.
Re:OpenAI Tokenizer by dqbd on Sunday April 09, 2023 @05:21PM (#35506594)
Author here, you are correct! The issue here is due to the fact that a single user-perceived character might span into multiple tokens. This should be fixed now.
Re:OpenAI Tokenizer by egorfine on Monday April 10, 2023 @05:00AM (#35510440)
Hey. Thank you! However has the fix not been deployed yet? Still shows broken UTF-8.

> a single user-perceived character might span into multiple tokens

Is this the way it works as designed or is this a bug?