News for hackers. Stuff that matters.
Re:OpenAI Tokenizer by egorfine on Thursday April 06, 2023 @10:49AM (#35469053)
Hey it seems that UTF-8 support is broken on the page.

Test phrase could be something like "Жизнь прекрасна и удивительна" ("Life is great" in russian).

I make an assumption that this is the implementation on the page that is broken, not the actual tokenizer. The reason: russian works perfectly in GPT-3 which I guess wouldn't be the case with a tokenization as presented on the page.

Re:OpenAI Tokenizer by dqbd on Sunday April 09, 2023 @05:21PM (#35506594)
Author here, you are correct! The issue here is due to the fact that a single user-perceived character might span into multiple tokens. This should be fixed now.
Re:OpenAI Tokenizer by egorfine on Monday April 10, 2023 @05:00AM (#35510440)
Hey. Thank you! However has the fix not been deployed yet? Still shows broken UTF-8.

> a single user-perceived character might span into multiple tokens

Is this the way it works as designed or is this a bug?