News for hackers. Stuff that matters.
Re:OpenAI Tokenizer by egorfine on Monday April 10, 2023 @05:00AM (#35510440)
Hey. Thank you! However has the fix not been deployed yet? Still shows broken UTF-8.

> a single user-perceived character might span into multiple tokens

Is this the way it works as designed or is this a bug?

There are no comments on this comment yet.